2024-09-11 17:14:46.494 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:14:46.497 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:14:46.497198816+08:00 2024-09-11 17:14:46.498 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:14:46.497+08:00 2024-09-11 17:14:46.499 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:14:46.499863008+08:00 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:14:46.497+08:00 and status AVAILABLE 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:14:46.500+08:00 2024-09-11 17:14:46.502 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:14:46.500+08:00 and status AVAILABLE 2024-09-11 17:14:46.504 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:46.509 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:14:46.510 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:46.512 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:14:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.518 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.537 INFO 2981023 --- [ocp-schedules-19,28729377afba161e,844898e7c11ada9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:46.677 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:14:46.678 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:14:46.681 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:14:46.678390360+08:00 2024-09-11 17:14:46.681 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:14:46.683 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:14:46.681+08:00 2024-09-11 17:14:46.685 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:14:46.685 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:14:46.687 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:14:46.681+08:00 and status AVAILABLE 2024-09-11 17:14:46.786 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:14:46.800 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:14:46.808 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:14:46.950 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.971 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.974 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.979 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:14:46.979 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:46.987 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:46.987 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:14:46.989 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:46.990 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.993 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:14:46.997 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:14:46.998 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:14:46.999 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.000 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.000 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.001 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.007 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:14:47.010 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:14:47.010 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:14:47.022 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.024 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.024 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:14:47.039 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:14:47.039 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:14:47.039 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:14:47.042 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:14:47.052 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:14:47.057 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:14:47.075 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:14:47.084 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:47.090 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.091 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.091 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:47.092 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:14:47.092 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.095 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:14:47.097 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:14:47.101 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.102 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.110 INFO 2981023 --- [ocp-async-3,e744b63972b3ab6e,bdf9c4ae66ff9266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:47.115 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.116 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.117 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.117 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:14:47.118 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.118 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.121 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.121 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.121 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.121 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.122 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:47.128 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:14:47.129 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:14:46.339035+08:00 2024-09-11 17:14:47.131 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.134 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:14:47.134 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.135 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:14:47.136 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:14:47.136 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:14:47.138 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:14:47.139 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:14:47.142 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.146 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.146 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:14:47.146 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:14:47.147 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:14:47.153 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:14:47.159 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:14:47.159 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.159 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:14:47.166 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:14:47.172 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:14:47.181 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:14:47.183 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:14:47.187 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.187 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:14:47.188 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.196 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:14:47.197 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.198 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.199 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.199 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:14:47.200 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:14:47.206 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.208 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:14:47.211 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:14:47.212 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:14:47.212 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:14:47.214 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:14:47.224 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.224 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.225 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.225 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.225 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.226 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.226 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:14:47.228 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:14:47.230 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:14:47.231 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:14:47.236 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:14:47.237 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:14:47.259 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:14:47.291 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:14:47.294 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.295 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.303 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:14:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8aa2d501a71e0b8,afce71eec56b17b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:47.314 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:14:47.315 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:14:47.315 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:14:47.319 INFO 2981023 --- [prometheus-minute-schedule-1,c8aa2d501a71e0b8,afce71eec56b17b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:47.331 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:14:47.343 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:14:47.361 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:14:47.369 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:14:47.399 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:14:47.410 INFO 2981023 --- [host-check-clockdiff2,078af027df141134,5fd2e71c63db7090] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046087 0 0 , error= 2024-09-11 17:14:47.411 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.420 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.426 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.436 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.437 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.439 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:14:47.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:14:47.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:14:47.469 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:14:47.479 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:14:47.481 INFO 2981023 --- [ocp-schedules-4,5841680a06b1154e,1bf0ecc97a7b41c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:47.730 INFO 2981023 --- [ocp-async-4,89c2727f785a9914,52565d1842ce4f8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:14:49.110 INFO 2981023 --- [ocp-async-8,ef87bddf6bae9c39,3edfc0d45a3d2de0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:49.455 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:14:49.455+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:14:49.458 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:14:49.497 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 17:14:49.498 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:14:49.510 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:49.512 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:49.685 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:14:49.686 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.724 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.725 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.727 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.740 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.750 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.755 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.758 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.758 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.759 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1241D724-0-0] 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.766 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.766 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.783 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.783 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.784 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.785 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.785 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.786 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.787 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1241D72E-0-0] 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.790 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.796 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.796 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.797 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.800 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.800 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.802 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:14:49.806 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.807 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.815 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.831 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.832 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:14:49.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.839 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.841 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.843 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.845 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:14:49.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:14:49.857 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:14:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:14:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:14:50.163 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.170 INFO 2981023 --- [ocp-schedules-21,7c6be46466fe6bac,6264d9ad62e79b96] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:14:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,53c5f12530e3e05d,b9c6a909e1bc8d85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:50.315 INFO 2981023 --- [prometheus-minute-schedule-1,53c5f12530e3e05d,b9c6a909e1bc8d85] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:51.110 INFO 2981023 --- [ocp-async-6,fa7a54c78f2ca956,b5d15864890e4da9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:51.479 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:14:51.480 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:14:51.480 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.480 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.481 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.481 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:51.546 INFO 2981023 --- [ocp-schedules-1,4e38c0db479d564b,ae883d0240cf2e7e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:14:51.630 INFO 2981023 --- [ocp-schedules-12,f062efd065cba489,aa436896fbd57475] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:14:51.647 WARN 2981023 --- [ocp-schedules-12,f062efd065cba489,aa436896fbd57475] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:14:52.223 INFO 2981023 --- [host-check-clockdiff3,078af027df141134,4698839c495fd1e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046092 1 1 , error= 2024-09-11 17:14:52.512 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:52.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:52.953 INFO 2981023 --- [ocp-schedules-8,cd2cbd297128d6f7,ad15d32ea7eddc97] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:14:52.953 INFO 2981023 --- [ocp-schedules-8,cd2cbd297128d6f7,ad15d32ea7eddc97] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:14:53.103 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:14:53.110 INFO 2981023 --- [ocp-async-2,5ca5f44ba560a162,006104bb5aee8769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:53.112 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:14:53.113 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:14:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a64724a98c76032,1e473cc0a676d656] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a64724a98c76032,1e473cc0a676d656] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:54.104 INFO 2981023 --- [ocp-schedules-28,99ef3b92f9ae72b2,07eaa6c60db4d656] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:14:55.111 INFO 2981023 --- [ocp-async-1,b9baa55b907379a6,96160a52161058cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:55.131 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:14:55.131 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:14:55.147 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:14:55.148 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:14:55.148 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:14:55.149 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:14:55.150 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:14:55.151 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:14:55.164 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.166 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:14:55.167 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:14:55.170 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:14:55.170 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:14:55.171 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.171 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.172 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:14:55.172 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.173 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:14:55.176 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:14:55.176 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:14:55.177 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:14:55.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:55.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:55.585 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:14:55.587 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:14:55.587 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:14:55.605 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:14:55.612 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:14:55.612 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:14:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,e1026f94f8058551,654d0eb81e0b3efd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,e1026f94f8058551,654d0eb81e0b3efd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:14:56.350607+08:00, lastProcessTime=2024-09-11T17:14:46.339035+08:00 2024-09-11 17:14:56.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:14:56.350607+08:00 2024-09-11 17:14:57.111 INFO 2981023 --- [ocp-async-9,3210ee291f3c716f,54656650a8cf1910] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:57.504 INFO 2981023 --- [ocp-schedules-16,312c9f9b6e51fdee,9576f8efc7ff628b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:14:57.564 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56} 2024-09-11 17:14:57.566 WARN 2981023 --- [scanner-13446614-1,312c9f9b6e51fdee,f35d1816ea6c479b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:14:57.569 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkpr 2024-09-11 17:14:57.570 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.579 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.615 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:14:57.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.673 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:14:57.719 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697719000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.719Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.723 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:14:57.723 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.730 INFO 2981023 --- [ocp-async-5,a3af490ecca20ef8,18ffeb84efdbd9c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:14:57.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:14:57.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.719, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.736 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.739 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.782 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697782000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.782Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.782 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697782000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.782Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.785 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697785000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.785Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.791 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697791000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.791Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.792 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697792000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.792Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.793 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697793000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.793Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.796 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 56 2024-09-11 17:14:57.796 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.802 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697802000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.802Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.803 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697803000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.803Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.810 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.810 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.811 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.811 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.816 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.816 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.827 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 227 2024-09-11 17:14:57.827 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.828 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.828 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.829 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.829 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.830 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.830 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.831 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.831 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.832 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.832 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.834 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.834 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.837 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.837 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.842 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.842 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.846 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.846 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.847 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.782, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.785, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.782, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.853 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.861 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.791, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.864 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.864 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.867 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.867 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.792, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.793, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.876 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.878 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.880 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697880000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.880Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.883 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697883000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.883Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.885 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.802, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.888 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.803, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.890 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697890000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.890Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.897 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.897 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.900 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.900 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.911 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697911000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.911Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.913 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.913 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.917 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.917 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697924000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.924Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.926 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697926000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.926Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.928 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.928 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.935 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.935 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.937 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.937 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697937000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.937Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.939 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.940 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.88, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.941 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.941 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.942 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.942 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.943 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.945 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.945 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.947 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.883, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.948 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.948 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.949 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.949 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.89, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.952 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.952 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.957 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.958 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.958 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.911, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.962 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.962 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.965 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.924, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.965 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.926, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.968 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.937, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.971 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:14:58.175 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ad45983778f83f93,17b76a97476314f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ad45983778f83f93, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:14:58.197 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ad45983778f83f93,17b76a97476314f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ad45983778f83f93, duration=27 ms] 2024-09-11 17:14:58.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:58.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:59.111 INFO 2981023 --- [ocp-async-3,cedc6836e45155b9,c0e36afdd7c4db26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,14262f274542b7d5,4e4e425e65e3e9bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:59.315 INFO 2981023 --- [prometheus-minute-schedule-1,14262f274542b7d5,4e4e425e65e3e9bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:00.456 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:10:01.0, nextRun=2024-09-11 17:15:00.0, currentDate=Wed Sep 11 17:15:00 HKT 2024 2024-09-11 17:15:00.456 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:10:01.0, nextRunTime=2024-09-11 17:15:00.0} 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:15:00.460+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:15:00.472 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:10:01.0, nextRun=2024-09-11 17:15:00.0, currentDate=Wed Sep 11 17:15:00 HKT 2024 2024-09-11 17:15:00.472 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:10:01.0, nextRunTime=2024-09-11 17:15:00.0} 2024-09-11 17:15:00.485 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 17:15:00.491 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T17:15:00.492+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:15:00.525 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aec0913d1c45dc7a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048155, operation:EXECUTE, state: RUNNING 2024-09-11 17:15:00.535 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aec0913d1c45dc7a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048156, operation:EXECUTE, state: RUNNING 2024-09-11 17:15:00.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048155, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048175, task_operation=execute, latest_execution_start_time=2024-09-11T17:15:00.518+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048155}, listMap={}}, executor=10.31.35.10 2024-09-11 17:15:00.554 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048156, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048176, task_operation=execute, latest_execution_start_time=2024-09-11T17:15:00.534+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048156}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 17:15:00.556 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:15:00.561 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.575 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:15:00.578 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:15:00.593 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:15:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:15:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:15:00.610 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:15:00.611 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.615 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:15:00.627 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048156, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:15:00.687 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.735 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:15:00.736 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:15:00.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.823 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048176, status:SUCCESS 2024-09-11 17:15:00.826 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048176, status:SUCCESS 2024-09-11 17:15:00.859 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.912 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.931 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:00.946 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.010 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.020 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.047 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.054 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:15:01.054 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:15:01.060 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.115 INFO 2981023 --- [ocp-async-1,44177221a3fb30a8,fa4aeecef0ffd5c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:01.456 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.496 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.509 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:01.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:01.519 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.531 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.537 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.543 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:15:01.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.549 INFO 2981023 --- [ocp-schedules-7,830a0bebad6e5dbe,9e228a5b362139b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:01.566 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.575 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.586 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048155, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:15:01.838 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048175, status:SUCCESS 2024-09-11 17:15:01.839 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048175, status:SUCCESS 2024-09-11 17:15:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e6781b2a7b9825fd,a899b01b6e634728] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,e6781b2a7b9825fd,a899b01b6e634728] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:03.111 INFO 2981023 --- [ocp-async-6,c677a03b13280396,8fa19eaf551947bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:04.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:04.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:05.111 INFO 2981023 --- [ocp-async-10,3579462b9c823148,4887d245ff753672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,5ef39fffa7272850,7281d061c0d015bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,5ef39fffa7272850,7281d061c0d015bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:06.351739+08:00, lastProcessTime=2024-09-11T17:14:56.350607+08:00 2024-09-11 17:15:06.494 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:06.351739+08:00 2024-09-11 17:15:06.985 INFO 2981023 --- [ocp-schedules-21,87fc260bc103b8ab,4efc8c5eb66ed096] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:15:06.996 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:15:06.996+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:15:06.997 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:15:06.998 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:15:06.999 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:15:07.111 INFO 2981023 --- [ocp-async-3,e6ad6fa7b719bb84,8fc1349d6e374129] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:07.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:07.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:07.725 INFO 2981023 --- [ocp-schedules-11,b5e469a1e8677010,eeae6a17f80b81bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:07.731 INFO 2981023 --- [ocp-async-7,d426046d35e686c5,83ff9411592bb425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:07.763 WARN 2981023 --- [scanner-13447013-1,b5e469a1e8677010,f97da4213289fcef] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:07.807 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643345, data is empty. 2024-09-11 17:15:07.815 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:07.821 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:07.838 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643345, data is empty. 2024-09-11 17:15:07.838 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643345, data is empty. 2024-09-11 17:15:07.842 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:07.845 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:07.847 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:07.850 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:07.964 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:15:07.970 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:07.975 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,18e782c41e3749fe,0554d29e7ae5eabc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,18e782c41e3749fe,0554d29e7ae5eabc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:09.112 INFO 2981023 --- [ocp-async-5,9102f5ef663f3075,b4c0e7e490767d71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:15:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:15:10.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:10.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:15:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:15:11.112 INFO 2981023 --- [ocp-async-4,6f4c25ede2095d66,fa2d41399b9c43f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6acc61569a433d0b,df2c68c31b3a0f19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,6acc61569a433d0b,df2c68c31b3a0f19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:11.556 INFO 2981023 --- [ocp-schedules-4,49680e8223521139,ff8f7e0bdd5d25a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:13.112 INFO 2981023 --- [ocp-async-7,d46837ead8d239dc,8e9df53226ad02e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:13.116 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:13.126 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:13.126 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:13.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:13.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:13.590 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046040, nodeCount=480, affectRows=480 2024-09-11 17:15:13.753 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=49, affectRows=49 2024-09-11 17:15:13.753 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=47, validCount=47 2024-09-11 17:15:13.771 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=1000, affectRows=1000 2024-09-11 17:15:13.774 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=999, validCount=999 2024-09-11 17:15:13.819 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=1000, affectRows=1000 2024-09-11 17:15:13.821 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=1000, validCount=1000 2024-09-11 17:15:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0dd4f73ef56d4cb8,a9f808e3c04a0898] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,9d61113375506af2,750ca2e61e639ad7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:15:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,0dd4f73ef56d4cb8,a9f808e3c04a0898] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:15.112 INFO 2981023 --- [ocp-async-8,d5789919230a966d,5ad562eaac204375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:15.365 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046040, nodeCount=480, validCount=480 2024-09-11 17:15:16.021 INFO 2981023 --- [ocp-schedules-19,a457f1f152d73781,870508e664abefbf] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.021 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:16.022 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:16.023 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:16.024 INFO 2981023 --- [ocp-schedules-29,3782f9f9f3992e70,32f1a055308ab715] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff1,3782f9f9f3992e70,ba0c7e17acf422ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff2,3782f9f9f3992e70,b2aaf7501296a874] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff3,3782f9f9f3992e70,3538d0d89e1d0a3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:15:16.025 INFO 2981023 --- [host-check-clockdiff4,3782f9f9f3992e70,e9b60b30660f5095] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:15:16.034 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:16.052 INFO 2981023 --- [ocp-schedules-10,f606e9f8b52de1c0,69d4b03307872284] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:16.054 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:16.059 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.059 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.063 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.063 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-clockdiff4,3782f9f9f3992e70,e9b60b30660f5095] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-clockdiff2,3782f9f9f3992e70,b2aaf7501296a874] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.070 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.071 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.072 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.073 INFO 2981023 --- [host-check-clockdiff1,3782f9f9f3992e70,ba0c7e17acf422ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.077 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.100 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.100 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.183 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.187 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:15:16.188 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.189 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.190 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:15:16.190 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-6,fbd4b38e686ebef6,cb620410fd83ee32] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:15:16.209 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:16.263 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.264 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:15:16.277 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.281 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.390 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.392 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.396 INFO 2981023 --- [ocp-schedules-13,4b631d937f127bfe,5f887bcb096bc944] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:15:16.396 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:15:16.395337687+08:00 2024-09-11 17:15:16.400 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:15:16.396+08:00 2024-09-11 17:15:16.456 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:15:16.396+08:00 and status AVAILABLE 2024-09-11 17:15:16.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:16.458293+08:00, lastProcessTime=2024-09-11T17:15:06.351739+08:00 2024-09-11 17:15:16.466 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.468 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.469 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:15:16.469 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:15:16.471 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:15:16.471 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:15:16.471334653+08:00 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:15:16.473834171+08:00 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:15:16.471+08:00 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:15:16.475 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:15:16.474950060+08:00 2024-09-11 17:15:16.475 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:15:16.474+08:00 2024-09-11 17:15:16.476 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:15:16.475+08:00 2024-09-11 17:15:16.476 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:15:16.471+08:00 and status AVAILABLE 2024-09-11 17:15:16.476 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.477 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.478 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:15:16.478 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.480 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:15:16.480 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:16.482 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:15:16.474+08:00 and status AVAILABLE 2024-09-11 17:15:16.482 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:15:16.475+08:00 and status AVAILABLE 2024-09-11 17:15:16.486 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.492 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.501 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:15:16.519 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:15:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:16.522 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:16.524 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.524 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.526 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.560 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.562 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.563 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.564 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.572 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.575 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.590 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.592 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.593 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:15:16.595 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:16.610 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:15:16.615 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:15:16.617 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:15:16.627 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:15:16.628 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:15:16.630 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:15:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:15:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:16.659 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:15:16.660 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.666 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.666 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:16.674 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.682 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:15:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:16.697 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:15:16.702 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:15:16.702 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:15:16.711 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.712 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:15:16.713 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.714 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:15:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:16.720 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.721 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.721 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:15:16.722 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.725 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:15:16.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.732 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.742 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:15:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.744 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:15:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.745 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:16.748 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:16.756 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:15:16.757 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:15:16.758 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:15:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:16.768 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:15:16.772 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:15:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.782 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:15:16.786 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.787 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:15:16.791 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:15:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:16.819 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:15:16.829 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:15:16.835 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:15:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:16.849 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:16.458293+08:00 2024-09-11 17:15:16.851 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:16.854 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:16.854 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:15:16.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:16.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:15:16.859 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:17.112 INFO 2981023 --- [ocp-async-5,3125272fff6210a6,f30518e78a90e5e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b2274e9f067bd7a6,29c7e49043641c99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b2274e9f067bd7a6,29c7e49043641c99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:17.731 INFO 2981023 --- [ocp-async-8,7bc69515a95ebe21,e977ba75d54dd8a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:18.015 INFO 2981023 --- [ocp-schedules-2,d5af3c002443a0d8,c6ce52bc98630b47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:18.064 WARN 2981023 --- [scanner-13448830-1,d5af3c002443a0d8,b491fb4ec5a0a584] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:18.102 INFO 2981023 --- [alarm-detect-2,d5af3c002443a0d8,3ecd3615b227d52c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046085 ~ 1726046108 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046100, value=1.751)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.751, values={1726046100=1.751}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:15:19.112 INFO 2981023 --- [ocp-async-7,20f03ca3ee2598aa,97b3a21f6bbd482e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:19.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:19.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6c201668e92458d0,ab7af4988cd4ab9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,6c201668e92458d0,ab7af4988cd4ab9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:21.112 INFO 2981023 --- [ocp-async-6,a156e49c08bce1a6,0c01767d6d7b800a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:21.438 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:15:21.438 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:15:21.472 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:15:21.481 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:15:21.527 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b0087ed0fb50a108,a0cd5809aaa7dff7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=b0087ed0fb50a108, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:21.556 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b0087ed0fb50a108,a0cd5809aaa7dff7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=b0087ed0fb50a108, duration=34 ms] 2024-09-11 17:15:21.558 INFO 2981023 --- [ocp-schedules-14,0650b8266b4e02c0,0d0635ea1c2f137f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:21.732 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:15:21.903 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:21.903 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:15:21.909 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:21.910 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:21.911 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:21.912 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:21.912 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:21.913 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:21.914 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:21.914 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:21.935 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.942 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.945 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.946 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.948 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:15:22.002 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.557222096S 2024-09-11 17:15:22.002 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:15:22.010 INFO 2981023 --- [host-check-clockdiff3,3782f9f9f3992e70,3538d0d89e1d0a3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046122 1 1 , error= 2024-09-11 17:15:22.052 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:15:22.052 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:15:22.070 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:15:22.112 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:15:22.192 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.boot.SpringApplication : Started application in 0.75 seconds (JVM running for 81922.065) 2024-09-11 17:15:22.192 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.750769236S 2024-09-11 17:15:22.337 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: clock_diff_milliseconds, cached timestamp: 1726038916 2024-09-11 17:15:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = clock_diff_milliseconds 2024-09-11 17:15:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_available, cached timestamp: 1726038916 2024-09-11 17:15:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_available 2024-09-11 17:15:22.413 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:15:22.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:22.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:23.112 INFO 2981023 --- [ocp-async-5,dd719cf3b4ce5f0d,03cf26f3d5fcd65b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,918ce2a15b21c239,b81fc132f0aa6164] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,918ce2a15b21c239,b81fc132f0aa6164] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:24.106 INFO 2981023 --- [ocp-schedules-10,3d584cb67330a46e,0e133a1332224a78] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:15:25.112 INFO 2981023 --- [ocp-async-1,dd9b7117da28a5e6,ce18c50f419f4195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:25.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:25.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,1390a8b144c703d9,9aacfac3c55310ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,1390a8b144c703d9,9aacfac3c55310ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:26.337700+08:00, lastProcessTime=2024-09-11T17:15:16.458293+08:00 2024-09-11 17:15:26.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:26.337700+08:00 2024-09-11 17:15:27.113 INFO 2981023 --- [ocp-async-9,9695dc98a3ae6f7f,d28c4110bbb06387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:27.731 INFO 2981023 --- [ocp-async-7,11d2e98d8664b62e,9507751ff578e493] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:28.235 INFO 2981023 --- [ocp-schedules-27,4f43c1c9bb57f109,0f8fb5d514f64376] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:28.279 WARN 2981023 --- [scanner-13449070-1,4f43c1c9bb57f109,61a7483251a89ed9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:28.307 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.315 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.323 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.327 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.329 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.335 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.335 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.340 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.343 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.432 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.438 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.442 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:28.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:29.113 INFO 2981023 --- [ocp-async-5,72afa57d1014171a,024c535c79a63ff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,67e822a0efdb7132,880fab40bbec552d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,67e822a0efdb7132,880fab40bbec552d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:31.113 INFO 2981023 --- [ocp-async-4,53159b4d0b190545,b046f5fbf22ad2e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:31.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:31.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:31.561 INFO 2981023 --- [ocp-schedules-14,9f95cdb6e3dfb638,929e5eb10a8dad6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,72925cfe6b90bf16,a7204fbee707f9cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,72925cfe6b90bf16,a7204fbee707f9cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:33.113 INFO 2981023 --- [ocp-async-9,5c56f0404a7c0d11,4952b9864c3c7593] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:33.128 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:33.137 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:33.137 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:34.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:34.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:35.113 INFO 2981023 --- [ocp-async-2,5a05b5b5fd4a7a4a,91036f82aa993607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe22cc47d61cb77b,cb4ce551574a9212] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,fe22cc47d61cb77b,cb4ce551574a9212] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:36.231 INFO 2981023 --- [ocp-schedules-29,6fb645ed10f25783,6a49a98fd4ac7f61] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:15:36.248 WARN 2981023 --- [ocp-schedules-29,6fb645ed10f25783,6a49a98fd4ac7f61] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:15:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:36.353266+08:00, lastProcessTime=2024-09-11T17:15:26.337700+08:00 2024-09-11 17:15:36.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:36.353266+08:00 2024-09-11 17:15:36.935 INFO 2981023 --- [ocp-schedules-5,e78c0eee4b30853c,4f0ce08d9bdc2226] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:15:37.113 INFO 2981023 --- [ocp-async-1,f6257201ef04e3f3,b4729f19f10e74e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:37.314 INFO 2981023 --- [prometheus-cache-monitor1,abedbdf99b3c455c,b0788db7c32d4349] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:15:37.315 INFO 2981023 --- [prometheus-cache-monitor1,abedbdf99b3c455c,b0788db7c32d4349] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:15:37.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:37.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:37.731 INFO 2981023 --- [ocp-async-5,f4caf58bcc0e1618,5f26ef35006af425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4fd8d7d07d3ef3d,301e614abbdd8c49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4fd8d7d07d3ef3d,301e614abbdd8c49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:38.482 INFO 2981023 --- [ocp-schedules-9,9181966f8e336a31,52997034d750bc85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:38.525 WARN 2981023 --- [scanner-13449348-1,9181966f8e336a31,cdb3bedad62cdc98] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:39.113 INFO 2981023 --- [ocp-async-9,bf615885256afc97,e9621a2db5aff5a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:40.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:15:40.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:15:40.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:40.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:41.113 INFO 2981023 --- [ocp-async-8,eca4bfb2301b3294,b8becedcca84bb69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c5097d0d0ffbcdd,559d5fcab312b82e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,0c5097d0d0ffbcdd,559d5fcab312b82e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:41.564 INFO 2981023 --- [ocp-schedules-7,48fe7bc138ed57df,ae4f5e25b2d9f47a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:43.113 INFO 2981023 --- [ocp-async-10,606f50285b4a797e,d2b5e108dc9e9638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:43.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:43.533 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,03627dba0ba4c573,81ddf499139486e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,03627dba0ba4c573,81ddf499139486e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:45.101 INFO 2981023 --- [prometheus-second-schedule-1,2d1f333ec756bf51,0ed340125b2d971c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:15:45.114 INFO 2981023 --- [ocp-async-7,4be069b7b2a098ab,a55aa480391e0d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:46.014 INFO 2981023 --- [ocp-schedules-21,8b11a26c35d22dd0,8f2e0211f62c7260] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.014 INFO 2981023 --- [host-check-clockdiff1,8b11a26c35d22dd0,5f157f559ff81197] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:15:46.014 INFO 2981023 --- [host-check-clockdiff2,8b11a26c35d22dd0,20ceb8fcb24cd7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:15:46.015 INFO 2981023 --- [host-check-clockdiff4,8b11a26c35d22dd0,944540f6c1efb892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:15:46.015 INFO 2981023 --- [host-check-clockdiff3,8b11a26c35d22dd0,2a137a9c594d7c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:15:46.023 INFO 2981023 --- [ocp-schedules-6,59f055caaf67e8a5,dd1fd9471e63e53b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.028 INFO 2981023 --- [host-check-clockdiff1,8b11a26c35d22dd0,5f157f559ff81197] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.029 INFO 2981023 --- [ocp-schedules-23,2895f7beb4b1873c,6dd19d607f832135] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:46.031 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:46.038 INFO 2981023 --- [host-check-clockdiff4,8b11a26c35d22dd0,944540f6c1efb892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.044 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:46.044 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:46.048 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:46.048 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:46.050 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.054 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.056 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.056 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.058 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.058 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.061 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.064 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.066 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.067 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.104 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.104 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.118 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.165 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:15:46.177 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.191 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:15:46.193 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:15:46.195 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.198 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:15:46.198 INFO 2981023 --- [ocp-schedules-2,4c44d269f3d6dade,6ce9f7fcdcec8648] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:15:46.257 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.266 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.266 INFO 2981023 --- [ocp-schedules-27,a1cc40a920c14eb3,438bb0aa2208cfd3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:15:46.271 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.277 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.285 INFO 2981023 --- [ocp-schedules-1,f862140bdfd75496,b46618d3e8d79fff] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:15:46.286 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:46.304 INFO 2981023 --- [ocp-schedules-27,a1cc40a920c14eb3,438bb0aa2208cfd3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:15:46.307 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:15:46.368 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:15:46.380 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.387 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:15:46.388 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:46.387955+08:00, lastProcessTime=2024-09-11T17:15:36.353266+08:00 2024-09-11 17:15:46.389 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:15:46.389049432+08:00 2024-09-11 17:15:46.390 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:15:46.389+08:00 2024-09-11 17:15:46.393 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:15:46.391210575+08:00 2024-09-11 17:15:46.393 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:15:46.393+08:00 2024-09-11 17:15:46.396 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:15:46.389+08:00 and status AVAILABLE 2024-09-11 17:15:46.398 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:15:46.393+08:00 and status AVAILABLE 2024-09-11 17:15:46.402 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.402 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.453 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.453 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:15:46.454 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.458 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:15:46.455306545+08:00 2024-09-11 17:15:46.458 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.460 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:15:46.461 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.462 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:15:46.458+08:00 2024-09-11 17:15:46.463 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:46.463 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.466 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:15:46.458+08:00 and status AVAILABLE 2024-09-11 17:15:46.476 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.476 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.477 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.477 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.479 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:15:46.481 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:15:46.481460529+08:00 2024-09-11 17:15:46.482 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.483 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:15:46.481+08:00 2024-09-11 17:15:46.486 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.488 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:15:46.481+08:00 and status AVAILABLE 2024-09-11 17:15:46.493 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:15:46.496 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.501 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.503 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:15:46.503 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:15:46.504 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.511 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.515 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.515 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.515 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.520 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.520 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.533 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:46.544 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:15:46.556 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:46.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:46.575 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.582 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:15:46.583 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.595 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.596 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:15:46.597 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.597 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.608 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:46.610 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.611 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:15:46.621 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:15:46.621 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.624 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:46.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:46.626 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.639 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:15:46.671 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:15:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:46.684 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.685 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:15:46.705 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:15:46.706 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:15:46.708 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:15:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:15:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:46.720 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:15:46.724 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.726 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.731 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.735 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.748 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.755 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.758 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.759 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:15:46.763 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:15:46.773 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.773 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.773 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:15:46.780 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:15:46.788 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:15:46.789 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.796 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.797 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.801 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:15:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.803 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.810 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:15:46.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.813 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.816 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:46.819 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.826 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:46.826 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.829 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:46.834 INFO 2981023 --- [ocp-schedules-11,895763297b294cc7,c2ed514a43f30e3a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:15:46.834 INFO 2981023 --- [ocp-schedules-11,895763297b294cc7,c2ed514a43f30e3a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:15:46.837 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:15:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:46.855 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:15:46.862 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:15:46.863 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:15:46.867 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:46.904 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:15:46.905 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.928 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.930 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.931 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.931 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:15:46.931 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.932 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.933 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.950 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:15:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.959 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.960 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:15:46.971 INFO 2981023 --- [host-check-clockdiff2,8b11a26c35d22dd0,20ceb8fcb24cd7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.974 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:15:46.979 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:15:46.981 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.983 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.985 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:46.987 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:15:47.009 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:47.009 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:47.012 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.013 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.013 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:15:47.013 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:47.014 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:47.014 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.032 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:15:47.032 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:47.044 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:15:47.057 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:15:47.060 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,90dd321703bb24c9,6a1487ef76153afb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=90dd321703bb24c9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:47.073 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:15:47.073 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.075 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:15:47.078 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.082 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:47.095 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:15:47.109 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:47.109 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:15:47.114 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:15:47.114 INFO 2981023 --- [ocp-async-5,bb6da06dc9766248,109c578ee6e8ac18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:47.117 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:15:47.120 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:15:47.126 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.131 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:15:47.134 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.135 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:47.135 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:15:47.142 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:46.387955+08:00 2024-09-11 17:15:47.145 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,90dd321703bb24c9,6a1487ef76153afb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=90dd321703bb24c9, duration=94 ms] 2024-09-11 17:15:47.152 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:47.152 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:15:47.157 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,77aae23621b1cb32,b40a45ef2258f117] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,77aae23621b1cb32,b40a45ef2258f117] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:47.731 INFO 2981023 --- [ocp-async-6,32218adc5f2187cb,4a3161d3d698ac44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:48.662 INFO 2981023 --- [ocp-schedules-20,b07d38b39f3c255e,abada15bc49990d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:48.687 WARN 2981023 --- [scanner-13449581-1,b07d38b39f3c255e,452b9d0b067c8e24] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:48.755 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.761 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.764 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.772 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.776 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.776 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.781 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.781 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.786 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.891 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:48.898 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:48.911 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:49.114 INFO 2981023 --- [ocp-async-10,7aaf7fc041e6b144,7dd16e6df10ca951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:49.512 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:15:49.512+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:15:49.515 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:15:49.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:49.566 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:49.625 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:15:49.626 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:15:49.857 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:15:49.859 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:49.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.890 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.891 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.893 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.901 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:15:49.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.907 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:15:49.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.917 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.917 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE13B1D734-0-0] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.941 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.944 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.944 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:15:49.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.953 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.956 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:15:49.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE13B1D73E-0-0] 2024-09-11 17:15:49.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:15:49.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.969 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.969 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.974 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:15:49.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.976 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:49.977 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.978 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.979 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:49.990 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.990 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.992 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:15:49.997 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:49.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:15:49.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.000 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:50.001 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:50.003 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:50.006 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.007 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:15:50.008 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.008 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:50.013 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:50.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:50.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:50.030 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:15:50.173 INFO 2981023 --- [ocp-schedules-5,39080cf6cc304398,a5968418ced2f417] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:15:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ad054d2a893088b3,62540c0c0ba839a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,ad054d2a893088b3,62540c0c0ba839a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:51.114 INFO 2981023 --- [ocp-async-7,50a030ed9ca8ce50,2fb0ba9cc4f6ac44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:51.481 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:15:51.482 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.483 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:51.566 INFO 2981023 --- [ocp-schedules-1,1ad9808188b8f829,2ddcd093752e6cfb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:51.673 INFO 2981023 --- [ocp-schedules-4,5413a2ad708b13d9,910b879b3cf0155f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:15:51.690 WARN 2981023 --- [ocp-schedules-4,5413a2ad708b13d9,910b879b3cf0155f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:15:51.977 INFO 2981023 --- [host-check-clockdiff3,8b11a26c35d22dd0,2a137a9c594d7c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046151 1 1 , error= 2024-09-11 17:15:52.566 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:52.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:52.959 INFO 2981023 --- [ocp-schedules-16,c17a17bc02b7fa3f,de1a76b961cd4753] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:15:52.959 INFO 2981023 --- [ocp-schedules-16,c17a17bc02b7fa3f,de1a76b961cd4753] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:15:53.114 INFO 2981023 --- [ocp-async-6,9c0169099d197c9f,698957e72401714d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:53.139 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:53.148 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:53.148 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,fc72cc1b9a5ed7b3,c1f1218133237eb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,fc72cc1b9a5ed7b3,c1f1218133237eb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:54.109 INFO 2981023 --- [ocp-schedules-6,30f388162f58c44b,740590ab504a99de] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:15:54.603 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:15:54.603 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:15:54.621 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:15:55.114 INFO 2981023 --- [ocp-async-1,d67ae20b0ed54087,185a8651ae9b3bf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:55.177 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:15:55.177 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:15:55.202 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:15:55.205 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:15:55.205 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:15:55.223 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.223 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.224 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:15:55.224 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.225 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:15:55.225 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.227 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.227 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:15:55.228 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.228 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.229 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:15:55.230 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.234 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:15:55.234 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:15:55.234 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:15:55.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:55.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:55.587 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:15:55.587 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:15:55.590 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:15:55.590 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:15:55.592 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:15:55.592 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:15:55.612 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:15:55.622 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:15:55.622 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:15:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,dfbff7b247fc26f6,047d2be0bec62b10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,dfbff7b247fc26f6,047d2be0bec62b10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:56.352086+08:00, lastProcessTime=2024-09-11T17:15:46.387955+08:00 2024-09-11 17:15:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:56.352086+08:00 2024-09-11 17:15:57.114 INFO 2981023 --- [ocp-async-9,59a7946132c3f706,f4d90c2a572f8dc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:57.731 INFO 2981023 --- [ocp-async-3,8d62c607f10f523e,b3bac20ce64c474e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:58.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:58.570 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:58.953 INFO 2981023 --- [ocp-schedules-14,9b098878e1cb1d61,effdd520ea4f576a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:59.000 WARN 2981023 --- [scanner-13449873-1,9b098878e1cb1d61,c454452ab3e5e9a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:59.114 INFO 2981023 --- [ocp-async-6,178beb1f7880a3ec,51a6427e2a73a05d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:59.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6b96245ec14fdf0b,f3e6c5a31dc454f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=6b96245ec14fdf0b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:59.163 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6b96245ec14fdf0b,f3e6c5a31dc454f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=6b96245ec14fdf0b, duration=54 ms] 2024-09-11 17:15:59.297 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d68a734cd10c5cb,7f9440d2e17d27e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8d68a734cd10c5cb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:15:59.306 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d68a734cd10c5cb,7f9440d2e17d27e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8d68a734cd10c5cb, duration=12 ms] 2024-09-11 17:15:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,c68cd8ac3d9eaf62,2999d7e9326814c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,c68cd8ac3d9eaf62,2999d7e9326814c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:01.118 INFO 2981023 --- [ocp-async-1,a03422dc325d181b,24ca11fd0596a2be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:01.385 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9b5180f72f6a3233,9d34801ce4e89476] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=9b5180f72f6a3233, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:01.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9b5180f72f6a3233,9d34801ce4e89476] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=9b5180f72f6a3233, duration=19 ms] 2024-09-11 17:16:01.569 INFO 2981023 --- [ocp-schedules-4,4e807e7579a32221,1d03191796780cd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:01.570 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:01.571 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:01.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9dc595a6f572a3eb,9281e95b68a5d779] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=9dc595a6f572a3eb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:01.656 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9dc595a6f572a3eb,9281e95b68a5d779] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=9dc595a6f572a3eb, duration=5 ms] 2024-09-11 17:16:01.987 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:16:01.988 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:02.033 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:02.034 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:02.039 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:02.041 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:16:02.041 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000} 2024-09-11 17:16:02.047 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:16:02.050 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000221, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:16:02.050 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:16:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,46ed7042502c5919,083014143f8f3cd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,46ed7042502c5919,083014143f8f3cd8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:02.785 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 17:16:02.785 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 17:16:02.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 17:16:02.789 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000221, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:16:02.791 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 17:16:03.115 INFO 2981023 --- [ocp-async-3,a6a5a64b3b623be6,087570b6965ed2dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:03.185 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f94e313cc412084f,016e0aaefd9acc99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f94e313cc412084f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:03.195 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f94e313cc412084f,016e0aaefd9acc99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f94e313cc412084f, duration=13 ms] 2024-09-11 17:16:03.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b6f32bb9125eb4de,837619f7cf402b13] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b6f32bb9125eb4de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:03.477 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b6f32bb9125eb4de,837619f7cf402b13] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b6f32bb9125eb4de, duration=6 ms] 2024-09-11 17:16:03.671 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000221 2024-09-11 17:16:04.571 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:04.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:05.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1958b13a940e4965,03b4d8fa8b5b904d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=1958b13a940e4965, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:05.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1958b13a940e4965,03b4d8fa8b5b904d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=1958b13a940e4965, duration=13 ms] 2024-09-11 17:16:05.115 INFO 2981023 --- [ocp-async-5,cc1f7ef2dcb73695,ba72b068026a04b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,2a0dad5996b34f52,7eb06736489e5ee6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,2a0dad5996b34f52,7eb06736489e5ee6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:05.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,387a76dbae9aa39c,e19dca127fc982ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=387a76dbae9aa39c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:05.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,387a76dbae9aa39c,e19dca127fc982ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=387a76dbae9aa39c, duration=5 ms] 2024-09-11 17:16:05.983 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:05.985 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:16:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:16:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:16:06.009 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 17:16:06.009 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:16:06.009 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:16:06.012 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:06.012 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:06.013 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:06.014 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:16:06.062 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:13:41, start=2024-09-11T17:10:11, end=2024-09-11T17:12:41, updates=289, granule=120000ms, delay=210000ms 2024-09-11 17:16:06.068 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:13:41, target=2024-09-11T17:12:41, granule=120000ms, delay=60000ms) 2024-09-11 17:16:06.068 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:12:41, target=2024-09-11T17:00:51, granule=600000ms, delay=710000ms) 2024-09-11 17:16:06.069 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=84ms, period=120000ms 2024-09-11 17:16:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:06.351213+08:00, lastProcessTime=2024-09-11T17:15:56.352086+08:00 2024-09-11 17:16:06.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:06.351213+08:00 2024-09-11 17:16:06.999 INFO 2981023 --- [ocp-schedules-15,5a4e4ab8355bbf9f,1f10f03ab399c612] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:16:07.012 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:16:07.012+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:16:07.013 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:16:07.014 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:16:07.015 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:16:07.018 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,df2f96bde29266c0,8f6b741dd2a17040] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=df2f96bde29266c0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:07.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,df2f96bde29266c0,8f6b741dd2a17040] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=df2f96bde29266c0, duration=12 ms] 2024-09-11 17:16:07.115 INFO 2981023 --- [ocp-async-6,7543de848749d975,8cef745c4d766a22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:07.253 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7fdc63d372916954,954ce951f6c26ba3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7fdc63d372916954, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:07.256 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7fdc63d372916954,954ce951f6c26ba3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7fdc63d372916954, duration=5 ms] 2024-09-11 17:16:07.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:07.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:07.731 INFO 2981023 --- [ocp-async-4,6f8f07db81b92a1d,5a0f95bc6f52ba0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,78862a733afa9b6b,891a592d01e48c16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,78862a733afa9b6b,891a592d01e48c16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:08.864 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8869de07392740aa,7121c64c6708f601] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=8869de07392740aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:08.874 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8869de07392740aa,7121c64c6708f601] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=8869de07392740aa, duration=13 ms] 2024-09-11 17:16:09.115 INFO 2981023 --- [ocp-async-3,8c513a147a477bba,fa40382d192fbc3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:09.178 INFO 2981023 --- [ocp-schedules-23,9185b9c9e3c2d9c3,a98e05e2c2f01772] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:09.223 WARN 2981023 --- [scanner-13450121-1,9185b9c9e3c2d9c3,b21b0eb32a943679] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:09.253 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:16:09.256 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:09.261 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:09.279 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:16:09.281 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:16:09.282 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:09.285 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:09.287 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:09.289 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:09.317 INFO 2981023 --- [alarm-detect-19,9185b9c9e3c2d9c3,9f416d7fa465c44b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046130 ~ 1726046159 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046145, value=2.962)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.962, values={1726046145=2.962}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:09.398 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:16:09.402 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:09.406 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:10.002 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:16:10.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:16:10.458 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d7767ee493acc082,a965211a71c6b440] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=d7767ee493acc082, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:10.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d7767ee493acc082,a965211a71c6b440] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=d7767ee493acc082, duration=13 ms] 2024-09-11 17:16:10.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:10.573 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:10.689 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,5e38289a6645ac48,f7ee7a61693200ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=5e38289a6645ac48, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:10.692 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,5e38289a6645ac48,f7ee7a61693200ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=5e38289a6645ac48, duration=6 ms] 2024-09-11 17:16:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:16:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:16:11.115 INFO 2981023 --- [ocp-async-2,33b81b297ead258f,8f74d3f0cc7afb56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,520e53623b57b7e9,477d2e2c5222c8f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,520e53623b57b7e9,477d2e2c5222c8f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:11.572 INFO 2981023 --- [ocp-schedules-25,fca5b0729e803d6c,3eb90cfd2c81afde] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:12.158 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,41ac541771c14512,abfad251b962fb19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=41ac541771c14512, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:12.169 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,41ac541771c14512,abfad251b962fb19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=41ac541771c14512, duration=13 ms] 2024-09-11 17:16:12.488 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b3eea57453d714de,102d7976716c0304] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b3eea57453d714de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:12.490 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b3eea57453d714de,102d7976716c0304] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b3eea57453d714de, duration=5 ms] 2024-09-11 17:16:13.115 INFO 2981023 --- [ocp-async-6,56430d06a7608523,ac0f93c51eaa584d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:13.151 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:13.159 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:13.159 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:13.573 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:13.574 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:13.830 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=1000, affectRows=1000 2024-09-11 17:16:13.833 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=999, validCount=999 2024-09-11 17:16:13.853 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=48, affectRows=48 2024-09-11 17:16:13.853 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=46, validCount=46 2024-09-11 17:16:13.858 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=1000, affectRows=1000 2024-09-11 17:16:13.860 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=1000, validCount=1000 2024-09-11 17:16:13.992 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7869047f98083008,b341b532dd128d35] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7869047f98083008, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:14.003 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7869047f98083008,b341b532dd128d35] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7869047f98083008, duration=14 ms] 2024-09-11 17:16:14.258 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,eeccbcac1a310fc3,8acda2b0ad39bfca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=eeccbcac1a310fc3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:14.260 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,eeccbcac1a310fc3,8acda2b0ad39bfca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=eeccbcac1a310fc3, duration=5 ms] 2024-09-11 17:16:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,dde5e05d090b7694,6d77c20baf414082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,35f2cc3a1400d10e,8a4fa70a2971c698] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:16:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,dde5e05d090b7694,6d77c20baf414082] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:14.904 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046100, nodeCount=480, affectRows=480 2024-09-11 17:16:15.115 INFO 2981023 --- [ocp-async-9,36a23d8e25765a11,5a7ec70d6d0bf7d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:15.753 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2fb36e2d960ff2d2,96f6d5326c4d96e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=2fb36e2d960ff2d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:15.763 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2fb36e2d960ff2d2,96f6d5326c4d96e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=2fb36e2d960ff2d2, duration=13 ms] 2024-09-11 17:16:16.014 INFO 2981023 --- [ocp-schedules-26,97a3a4b122e58884,41b32df0ac0278ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff1,97a3a4b122e58884,0f4e605b477458aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff2,97a3a4b122e58884,2d92282dd026c4a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff3,97a3a4b122e58884,6954326b7b1a25cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:16:16.017 INFO 2981023 --- [host-check-clockdiff4,97a3a4b122e58884,e0ff3cdbe2f73466] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:16:16.026 INFO 2981023 --- [ocp-schedules-29,04403b2295dad6ae,db33ebf95892af3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:16.030 INFO 2981023 --- [host-check-clockdiff4,97a3a4b122e58884,e0ff3cdbe2f73466] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046176 0 0 , error= 2024-09-11 17:16:16.030 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:16.031 INFO 2981023 --- [ocp-schedules-30,a1c3c4b4640e2522,c7d07eb597b7ffe8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.031 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:16.034 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:16.039 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:16.039 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:16.065 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.071 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.074 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.077 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.077 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.079 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.080 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.081 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.104 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.106 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.167 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.195 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.196 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-2,89cdbffce203ec8c,57157a1872de8e5e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:16:16.282 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.284 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.372 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.374 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.374 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:16.380 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:16:16.392 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:16.395298+08:00, lastProcessTime=2024-09-11T17:16:06.351213+08:00 2024-09-11 17:16:16.405 INFO 2981023 --- [ocp-schedules-19,ecc7aadab6fb871d,8c52a747b53102ca] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:16:16.406 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:16:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.409 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:16:16.409297808+08:00 2024-09-11 17:16:16.410 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:16:16.409+08:00 2024-09-11 17:16:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:16:16.413 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:16:16.409+08:00 and status AVAILABLE 2024-09-11 17:16:16.413 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:16:16.415 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.415 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.416 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.416 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:16:16.417 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:16.418 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:16:16.418338333+08:00 2024-09-11 17:16:16.419 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:16:16.418+08:00 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.454 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:16:16.455 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:16:16.418+08:00 and status AVAILABLE 2024-09-11 17:16:16.459 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.460 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:16:16.460 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:16:16.460642994+08:00 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.466 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:16:16.460+08:00 2024-09-11 17:16:16.468 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:16:16.460+08:00 and status AVAILABLE 2024-09-11 17:16:16.470 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:16:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.474 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:16:16.470788989+08:00 2024-09-11 17:16:16.476 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:16:16.477 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:16:16.474+08:00 2024-09-11 17:16:16.480 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:16:16.474+08:00 and status AVAILABLE 2024-09-11 17:16:16.483 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:16:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.484 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.485 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.485 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.486 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.487 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.488 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:16:16.495 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.496 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.507 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.509 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.512 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.518 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.535 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.539 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:16:16.543 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:16:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.559 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:16:16.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:16.562 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:16:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:16.569 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:16:16.570 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:16:16.573 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:16:16.574 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:16.576 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:16.587 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:16:16.598 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:16:16.601 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.605 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.606 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:16.609 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:16.609 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:16.621 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:16:16.630 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:16:16.635 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:16:16.639 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:16:16.653 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f0f0feac2bea7d8,8226ea9c7b34bc60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=4f0f0feac2bea7d8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.666 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:16:16.666 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:16:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f0f0feac2bea7d8,8226ea9c7b34bc60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=4f0f0feac2bea7d8, duration=21 ms] 2024-09-11 17:16:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:16:16.673 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.675 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:16:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.682 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:16:16.689 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.693 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:16:16.694 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.694 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:16:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.704 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:16.710 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:16:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:16.721 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:16:16.724 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:16:16.733 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:16:16.733 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:16:16.735 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.760 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:16:16.769 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.769 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.770 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:16:16.775 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:16:16.778 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:16:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.781 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.785 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:16.395298+08:00 2024-09-11 17:16:16.799 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:16:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.806 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:16:16.830 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:16:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.836 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.836 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:16:16.845 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:17.032 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046100, nodeCount=480, validCount=480 2024-09-11 17:16:17.115 INFO 2981023 --- [ocp-async-5,28cbb53ce783108f,fcd20e6d39ca4831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,77c05cf8464f3c1b,b33e52c728c384a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:17.318 INFO 2981023 --- [prometheus-minute-schedule-1,77c05cf8464f3c1b,b33e52c728c384a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:17.731 INFO 2981023 --- [ocp-async-3,775b59c41dea89d5,22402f10187829da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:17.810 INFO 2981023 --- [host-check-clockdiff2,97a3a4b122e58884,2d92282dd026c4a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046177 0 0 , error= 2024-09-11 17:16:18.375 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,736f74268e2be753,a996b6754f99ebdb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=736f74268e2be753, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:18.398 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,736f74268e2be753,a996b6754f99ebdb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=736f74268e2be753, duration=28 ms] 2024-09-11 17:16:18.670 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0847a1eb4a2fe13,d9ebaf5950f537de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b0847a1eb4a2fe13, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:18.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0847a1eb4a2fe13,d9ebaf5950f537de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b0847a1eb4a2fe13, duration=9 ms] 2024-09-11 17:16:19.115 INFO 2981023 --- [ocp-async-1,272995077aa42b5a,3675d4989a520f5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:19.448 INFO 2981023 --- [ocp-schedules-12,92f9ebfdef77a83f,b3c36114608811bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:19.508 WARN 2981023 --- [scanner-13451968-1,92f9ebfdef77a83f,26e0e3a7d490683c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:19.577 INFO 2981023 --- [alarm-detect-17,92f9ebfdef77a83f,0964e75d002bb351] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046145 ~ 1726046169 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046160, value=2.325)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.325, values={1726046160=2.325}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:19.577 INFO 2981023 --- [alarm-detect-17,92f9ebfdef77a83f,0964e75d002bb351] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046145 ~ 1726046169 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046160, value=1.706)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.706, values={1726046160=1.706}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:19.577 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:19.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:19.722 INFO 2981023 --- [host-check-clockdiff1,97a3a4b122e58884,0f4e605b477458aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046179 0 0 , error= 2024-09-11 17:16:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,aefd9552f7b21502,2d5c6e421f6d70a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,aefd9552f7b21502,2d5c6e421f6d70a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:20.407 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a08c653e1a00a207,e88478209781858b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=a08c653e1a00a207, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:20.417 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a08c653e1a00a207,e88478209781858b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=a08c653e1a00a207, duration=12 ms] 2024-09-11 17:16:21.115 INFO 2981023 --- [ocp-async-8,d5b1de8868f67fa9,8313a92f41a9d26c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:21.576 INFO 2981023 --- [ocp-schedules-22,78cdfc9fa8e114c1,764491b4eea8c299] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:21.951 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:21.951 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:16:21.956 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:21.956 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:21.957 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:21.960 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:21.961 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:21.961 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:21.961 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:21.964 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:21.987 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.987 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.992 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.993 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:16:22.013 INFO 2981023 --- [host-check-clockdiff3,97a3a4b122e58884,6954326b7b1a25cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046182 1 1 , error= 2024-09-11 17:16:22.166 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2c9a7de9eb04ef8d,8c4a4303490ad7e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=2c9a7de9eb04ef8d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:22.176 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2c9a7de9eb04ef8d,8c4a4303490ad7e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=2c9a7de9eb04ef8d, duration=13 ms] 2024-09-11 17:16:22.439 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,eb9f35ce4547f4f5,ea5fe17cc9f81847] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=eb9f35ce4547f4f5, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:22.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,eb9f35ce4547f4f5,ea5fe17cc9f81847] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=eb9f35ce4547f4f5, duration=5 ms] 2024-09-11 17:16:22.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:22.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:23.116 INFO 2981023 --- [ocp-async-7,0403c7a249e78315,9cd8ef8890e779e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a76dd42a49bb3c0a,1b9ea3407b026a87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,a76dd42a49bb3c0a,1b9ea3407b026a87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:24.112 INFO 2981023 --- [ocp-schedules-1,6d8238d1e8ed60be,7e2c40181c273c71] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:16:24.459 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7c89cb34151f7855,730061f16e44e262] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7c89cb34151f7855, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:24.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7c89cb34151f7855,730061f16e44e262] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7c89cb34151f7855, duration=13 ms] 2024-09-11 17:16:24.846 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8728fbc85b543b9c,0a38446264e883e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8728fbc85b543b9c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:24.848 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8728fbc85b543b9c,0a38446264e883e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8728fbc85b543b9c, duration=5 ms] 2024-09-11 17:16:25.116 INFO 2981023 --- [ocp-async-6,cf450cfd721af1a6,cdb74c1a0fa88858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:25.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:25.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,c379fd6990e8d000,c40605c5754a14f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,c379fd6990e8d000,c40605c5754a14f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:26.358 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:26.357214+08:00, lastProcessTime=2024-09-11T17:16:16.395298+08:00 2024-09-11 17:16:26.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:26.357214+08:00 2024-09-11 17:16:26.584 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e3465d0c138117da,ca7a2dd8a33797fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=e3465d0c138117da, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:26.595 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e3465d0c138117da,ca7a2dd8a33797fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=e3465d0c138117da, duration=14 ms] 2024-09-11 17:16:26.897 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,988ac4830c337be6,b44c94d5242fa7e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=988ac4830c337be6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:26.899 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,988ac4830c337be6,b44c94d5242fa7e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=988ac4830c337be6, duration=5 ms] 2024-09-11 17:16:27.116 INFO 2981023 --- [ocp-async-10,a142e5da68887622,78a4c89f11f57ae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:27.732 INFO 2981023 --- [ocp-async-1,f2de69171ded8a24,8799fedc099e345d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:28.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fbe326dcab22d239,5e5cd0e0ce407b58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=fbe326dcab22d239, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:28.511 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fbe326dcab22d239,5e5cd0e0ce407b58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=fbe326dcab22d239, duration=13 ms] 2024-09-11 17:16:28.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:28.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:28.773 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71ca981096374ef3,a6e70af85c48ca32] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=71ca981096374ef3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:28.775 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71ca981096374ef3,a6e70af85c48ca32] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=71ca981096374ef3, duration=5 ms] 2024-09-11 17:16:29.116 INFO 2981023 --- [ocp-async-7,877fe94451f8f2e2,90b62d861c8c50da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,57a16983e237ab1e,4580d6256d82e585] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,57a16983e237ab1e,4580d6256d82e585] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:29.649 INFO 2981023 --- [ocp-schedules-12,80db34e1913a17cb,d5855b5b25cb067d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:29.695 WARN 2981023 --- [scanner-13452161-1,80db34e1913a17cb,5eb3653f21e1003d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:29.723 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.728 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.733 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.741 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.745 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.748 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.752 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.753 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.757 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.865 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:29.868 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:29.872 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:30.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ed6caf3145304b18,571813d45091b771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ed6caf3145304b18, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:30.383 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ed6caf3145304b18,571813d45091b771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ed6caf3145304b18, duration=12 ms] 2024-09-11 17:16:30.632 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,44c8f5bf0fcd3360,e1950a02ef0caefa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=44c8f5bf0fcd3360, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:30.634 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,44c8f5bf0fcd3360,e1950a02ef0caefa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=44c8f5bf0fcd3360, duration=5 ms] 2024-09-11 17:16:31.116 INFO 2981023 --- [ocp-async-4,e61b797bc83c0f83,305f091f6dbb4b47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:31.579 INFO 2981023 --- [ocp-schedules-2,8dee53b25a0f8f72,6801e76f08b4ede4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:31.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:31.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:32.272 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,eaa6993608def31c,53e1ef5d82392344] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=eaa6993608def31c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:32.282 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,eaa6993608def31c,53e1ef5d82392344] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=eaa6993608def31c, duration=13 ms] 2024-09-11 17:16:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,6a386013c3d07ed7,e4c48f14c75f91ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,6a386013c3d07ed7,e4c48f14c75f91ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:33.116 INFO 2981023 --- [ocp-async-9,34c83986622075f4,db3ee251abde94fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:33.162 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:33.171 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:33.171 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:34.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,359d1d3c7eafca68,a12743a6a51e98e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=359d1d3c7eafca68, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:34.075 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,359d1d3c7eafca68,a12743a6a51e98e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=359d1d3c7eafca68, duration=12 ms] 2024-09-11 17:16:34.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:34.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:34.826 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f4fcfd21b8822592,5927530ea6d1bc0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=f4fcfd21b8822592, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:34.829 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f4fcfd21b8822592,5927530ea6d1bc0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=f4fcfd21b8822592, duration=5 ms] 2024-09-11 17:16:35.116 INFO 2981023 --- [ocp-async-3,33848c9b09204228,4ce6e12d7505bc6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,913555b8553d1a38,dd92cde6213ca73e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,913555b8553d1a38,dd92cde6213ca73e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:36.337234+08:00, lastProcessTime=2024-09-11T17:16:26.357214+08:00 2024-09-11 17:16:36.399 INFO 2981023 --- [ocp-schedules-22,1d586c71cecd02dd,ec070cbf5e2cb3b7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:16:36.422 WARN 2981023 --- [ocp-schedules-22,1d586c71cecd02dd,ec070cbf5e2cb3b7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:16:36.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:36.337234+08:00 2024-09-11 17:16:36.877 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f86ae446c7e44dc9,d32ee2f05186a1f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f86ae446c7e44dc9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:36.887 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f86ae446c7e44dc9,d32ee2f05186a1f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f86ae446c7e44dc9, duration=12 ms] 2024-09-11 17:16:37.116 INFO 2981023 --- [ocp-async-2,a042c66defe1a295,d8a808efe82a0e89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:37.151 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7f2473a089a5b8da,fd1d2d80dbbe3a78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7f2473a089a5b8da, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:37.154 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7f2473a089a5b8da,fd1d2d80dbbe3a78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7f2473a089a5b8da, duration=5 ms] 2024-09-11 17:16:37.314 INFO 2981023 --- [prometheus-cache-monitor1,cb53c58c702f4ab2,f623ccba0e0bac2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:16:37.315 INFO 2981023 --- [prometheus-cache-monitor1,cb53c58c702f4ab2,f623ccba0e0bac2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:16:37.410 INFO 2981023 --- [ocp-schedules-15,2984d6730d6f4343,35c495e097b9b226] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:16:37.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:37.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:37.732 INFO 2981023 --- [ocp-async-7,f00987b33179e3dd,03bb30f0cb674cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,819774dd24830b63,2de6e27b378a34cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,819774dd24830b63,2de6e27b378a34cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:38.813 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa020f15c7cf5a6e,fa97aa8130471ac6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=aa020f15c7cf5a6e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:38.824 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa020f15c7cf5a6e,fa97aa8130471ac6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=aa020f15c7cf5a6e, duration=14 ms] 2024-09-11 17:16:39.116 INFO 2981023 --- [ocp-async-1,4425d65e4b9dc2a8,bd75bab613336f61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:39.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,aa88213dc94d63e0,0b241ea0b8f4e2b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=aa88213dc94d63e0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:39.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,aa88213dc94d63e0,0b241ea0b8f4e2b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=aa88213dc94d63e0, duration=5 ms] 2024-09-11 17:16:39.911 INFO 2981023 --- [ocp-schedules-23,cc7a395370811e36,d9d0f224ddbbc2ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:39.954 WARN 2981023 --- [scanner-13452477-1,cc7a395370811e36,8a89ff65584710c9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:40.008 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:16:40.009 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046160 ~ 1726046189 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046175, value=1.955)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.955, values={1726046175=1.955}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-3] 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-3, alarm_template_id=0:host=10.31.35.7-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 17:16:40.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:40.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:40.802 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f3f9afcb8e4c02a5,551730e6a1fcbcc7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f3f9afcb8e4c02a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:40.812 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f3f9afcb8e4c02a5,551730e6a1fcbcc7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f3f9afcb8e4c02a5, duration=13 ms] 2024-09-11 17:16:41.084 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,13c75876056d5860,c1d8cc941e980016] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=13c75876056d5860, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:41.086 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,13c75876056d5860,c1d8cc941e980016] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=13c75876056d5860, duration=5 ms] 2024-09-11 17:16:41.116 INFO 2981023 --- [ocp-async-8,d817802a7bdf8139,59f452b9135699f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,78e284f89fc17cc4,b80038ca34e7ccda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,78e284f89fc17cc4,b80038ca34e7ccda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:41.582 INFO 2981023 --- [ocp-schedules-28,04176ff13a0fd9ab,7b30cf0541f7da8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:42.804 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d8d33c2be30f6765,1e06257e573f67b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=d8d33c2be30f6765, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:42.824 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d8d33c2be30f6765,1e06257e573f67b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=d8d33c2be30f6765, duration=23 ms] 2024-09-11 17:16:43.117 INFO 2981023 --- [ocp-async-6,1177f4ff89aa017f,8325e7a3cda9f5d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:43.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,585f18eed84e6c3e,ae3d8985957db465] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=585f18eed84e6c3e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:43.150 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,585f18eed84e6c3e,ae3d8985957db465] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=585f18eed84e6c3e, duration=6 ms] 2024-09-11 17:16:43.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:43.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,d1785da55b503961,292a6cfeca2299d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,d1785da55b503961,292a6cfeca2299d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:45.101 INFO 2981023 --- [prometheus-second-schedule-1,32f0909e4016f533,75ee45a07b67cd10] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:16:45.117 INFO 2981023 --- [ocp-async-4,c473cd7d576e7359,4ee2aca56caf3a97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:45.559 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,feac42dacbac723d,2822b512a300d220] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=feac42dacbac723d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:45.569 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,feac42dacbac723d,2822b512a300d220] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=feac42dacbac723d, duration=13 ms] 2024-09-11 17:16:45.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,614434d87186d8ea,38a0110309c9e953] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=614434d87186d8ea, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:45.825 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,614434d87186d8ea,38a0110309c9e953] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=614434d87186d8ea, duration=6 ms] 2024-09-11 17:16:46.015 INFO 2981023 --- [ocp-schedules-12,fc31e50b7280b1a6,8b0c7c7bc56ec92e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff1,fc31e50b7280b1a6,14cad3233dc2e63a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff2,fc31e50b7280b1a6,4211805bc0cdf6d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff3,fc31e50b7280b1a6,96f3dfa91eb4bc0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:16:46.018 INFO 2981023 --- [host-check-clockdiff4,fc31e50b7280b1a6,c966db7f1f91556a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:16:46.022 INFO 2981023 --- [ocp-schedules-22,cfd8cfc5a1e185ce,8fd0744bb55edceb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.026 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-clockdiff4,fc31e50b7280b1a6,c966db7f1f91556a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046206 0 0 , error= 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:46.035 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.038 INFO 2981023 --- [ocp-schedules-14,c15d1ac1a5fb60c5,ee5fa4112a1d0e02] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:46.040 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.041 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:46.042 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.043 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.044 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.044 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.045 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.046 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.046 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:46.047 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.048 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.049 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.052 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.052 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.054 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.058 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.063 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.065 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:16:46.067 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:16:46.067592407+08:00 2024-09-11 17:16:46.070 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:16:46.067+08:00 2024-09-11 17:16:46.071 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.072 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:16:46.067+08:00 and status AVAILABLE 2024-09-11 17:16:46.076 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:16:46.076 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.078 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:16:46.078 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:16:46.078026314+08:00 2024-09-11 17:16:46.079 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:16:46.078+08:00 2024-09-11 17:16:46.081 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:16:46.078+08:00 and status AVAILABLE 2024-09-11 17:16:46.083 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.084 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.085 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.092 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:16:46.080860564+08:00 2024-09-11 17:16:46.093 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:16:46.092+08:00 2024-09-11 17:16:46.093 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.094 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:16:46.095 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.102 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:16:46.092+08:00 and status AVAILABLE 2024-09-11 17:16:46.106 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:16:46.106257528+08:00 2024-09-11 17:16:46.107 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:16:46.106+08:00 2024-09-11 17:16:46.109 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:16:46.106+08:00 and status AVAILABLE 2024-09-11 17:16:46.115 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.117 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.126 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.132 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:16:46.138 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.139 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.148 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.150 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:16:46.166 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:16:46.169 INFO 2981023 --- [ocp-schedules-24,a214188a82b59a2f,74810e6c10e945e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:16:46.171 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:16:46.176 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.184 INFO 2981023 --- [ocp-schedules-24,a214188a82b59a2f,74810e6c10e945e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:16:46.187 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:16:46.187 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:16:46.188 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:16:46.189 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.190 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.191 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.194 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:16:46.195 INFO 2981023 --- [ocp-schedules-6,ba810cd98d4e07d5,f1eb3606d11bad04] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:16:46.195 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.197 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.198 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.199 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:46.209 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:16:46.211 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.216 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:16:46.216 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:16:46.226 INFO 2981023 --- [ocp-schedules-26,2110fe6ffc8a4c54,3dcbc5b207cd991b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:16:46.226 INFO 2981023 --- [ocp-schedules-26,2110fe6ffc8a4c54,3dcbc5b207cd991b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:16:46.228 INFO 2981023 --- [ocp-schedules-13,eacf4010d64a33e1,d1fdffbac5fb3b8d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:16:46.238 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:16:46.257 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:16:46.279 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.282 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.283 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.284 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.284 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.285 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:16:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:16:46.288 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.289 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.289 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:16:46.291 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.291 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.294 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:16:46.305 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:16:46.305 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,14dcb2710686bab4,a38b4af499a6797c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=14dcb2710686bab4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:16:46.305 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,14dcb2710686bab4,a38b4af499a6797c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=14dcb2710686bab4, duration=0 ms] 2024-09-11 17:16:46.307 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.307 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.318 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.319 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.319 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.322 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:46.328 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:16:46.328 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96de2e2f83f1b4d0,d2f4945b935181ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=96de2e2f83f1b4d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:16:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96de2e2f83f1b4d0,d2f4945b935181ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=96de2e2f83f1b4d0, duration=1 ms] 2024-09-11 17:16:46.337 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:46.338 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.340 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:16:46.345 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:46.347806+08:00, lastProcessTime=2024-09-11T17:16:36.337234+08:00 2024-09-11 17:16:46.348 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:16:46.350 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.352 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:16:46.352 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.352 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:16:46.352 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.353 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.355 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.355 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.367 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.373 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:16:46.373 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:46.379 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.379 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:16:46.383 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:16:46.384 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:16:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:46.409 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.412 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.412 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.413 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.417 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:16:46.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:46.420 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:46.431 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:16:46.437 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.443 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.444 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:16:46.444 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:16:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.449 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.450 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.452 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.453 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.453 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.460 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.461 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:16:46.465 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:16:46.466 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:16:46.468 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:16:46.478 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:16:46.480 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:16:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.481 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:16:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:46.491 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:16:46.498 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:46.514 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:16:46.524 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:16:46.530 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:16:46.530 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:16:46.531 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:16:46.531 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:16:46.541 INFO 2981023 --- [pool-10-thread-7,0857941388d70e85,5fd44c6bf5620d50] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:16:46.541 INFO 2981023 --- [pool-10-thread-8,0857941388d70e85,e063a5202097be7b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:16:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.543 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.545 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:46.546 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.549 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:46.553 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:16:46.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:16:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:16:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:16:46.565 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:16:46.571 INFO 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:16:46.571 INFO 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:16:46.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:46.587 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.589 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:46.589 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:16:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:46.608 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.609 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.609 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.611 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.611 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.612 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.612 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.613 WARN 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:16:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.616 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.617 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:16:46.619 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.619 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:16:46.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.645 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.648 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.648 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:16:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.652 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:16:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.655 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:16:46.656 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:16:46.656 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:16:46.657 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.657 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.658 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:16:46.659 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.667 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:16:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.671 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:16:46.675 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.676 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.680 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:16:46.680 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:16:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.682 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.683 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.684 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.684 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.685 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.685 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.685 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.689 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:16:46.690 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:16:46.691 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.701 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:16:46.702 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:16:46.703 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.704 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.705 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:16:46.705 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:16:46.706 INFO 2981023 --- [pool-10-thread-8,0857941388d70e85,e063a5202097be7b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:16:46.707 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.709 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:46.714 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.719 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.719 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:16:46.723 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:16:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:16:46.729 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.733 INFO 2981023 --- [pool-10-thread-7,0857941388d70e85,5fd44c6bf5620d50] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:16:46.734 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.735 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.735 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.736 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.736 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.753 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:16:46.764 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:16:46.773 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.789 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:16:46.806 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:16:46.818 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:16:46.821 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 17:16:46.823 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:16:46.829 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:46.347806+08:00 2024-09-11 17:16:46.858 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:16:46.860 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:16:47.117 INFO 2981023 --- [ocp-async-5,16af094555db13bf,f75c5f55a1f124a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,9e5da42541590711,023679812bc61dc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:47.318 INFO 2981023 --- [prometheus-minute-schedule-1,9e5da42541590711,023679812bc61dc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:47.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,626664bd9897bcc0,141355fdad33067a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=626664bd9897bcc0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:47.494 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,626664bd9897bcc0,141355fdad33067a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=626664bd9897bcc0, duration=27 ms] 2024-09-11 17:16:47.732 INFO 2981023 --- [ocp-async-3,e83a5dcbe7708102,395b04e7a816e0ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:47.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa8ca79bc11f97e3,6a51f6f076d1bc81] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=aa8ca79bc11f97e3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:47.760 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa8ca79bc11f97e3,6a51f6f076d1bc81] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=aa8ca79bc11f97e3, duration=9 ms] 2024-09-11 17:16:47.807 INFO 2981023 --- [host-check-clockdiff2,fc31e50b7280b1a6,4211805bc0cdf6d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046207 0 0 , error= 2024-09-11 17:16:49.117 INFO 2981023 --- [ocp-async-9,9862bc2bb732800d,810dc954605dfae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:49.433 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,88d10bda729cb70e,503e568a55a0b3d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=88d10bda729cb70e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:49.443 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,88d10bda729cb70e,503e568a55a0b3d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=88d10bda729cb70e, duration=12 ms] 2024-09-11 17:16:49.589 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:49.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:49.648 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:16:49.648+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:16:49.650 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:16:49.695 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 48 to SqlAggregateData 2024-09-11 17:16:49.697 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:16:49.727 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ac1c8cf17022d373,279ea37e12629f22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=ac1c8cf17022d373, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:49.729 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ac1c8cf17022d373,279ea37e12629f22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=ac1c8cf17022d373, duration=5 ms] 2024-09-11 17:16:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:16:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:16:50.030 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:16:50.032 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:50.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.066 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.068 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.069 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.071 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.072 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.087 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.087 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.093 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.099 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE14D1D73E-0-0] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.100 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.100 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.115 INFO 2981023 --- [ocp-schedules-21,6d269dda202190be,3bc2637c5e024f87] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:50.121 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.121 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.128 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE14D1D748-0-0] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:16:50.132 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.133 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.135 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.136 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.139 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.140 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.141 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.141 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.142 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.144 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.144 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:16:50.149 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.150 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:16:50.152 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.154 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.156 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.172 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.178 INFO 2981023 --- [ocp-schedules-3,b65fbb8c95eaf294,9d603b6a04a7904a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:16:50.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:16:50.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.184 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.187 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:16:50.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:16:50.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.190 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.196 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:50.200 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:50.213 WARN 2981023 --- [scanner-13452658-1,6d269dda202190be,827a47ebaad58651] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:50.233 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:16:50.236 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:50.242 INFO 2981023 --- [host-check-clockdiff1,fc31e50b7280b1a6,14cad3233dc2e63a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046210 0 0 , error= 2024-09-11 17:16:50.246 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:16:50.251 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:16:50.267 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:50.269 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:50.270 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:16:50.274 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:16:50.274 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:16:50.279 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:16:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f1c5e66be3575c1b,5fa19965e398e753] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:50.318 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,49787abe6320d3fb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046175 ~ 1726046200 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046190, value=1.522)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.522, values={1726046190=1.522}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:50.324 INFO 2981023 --- [prometheus-minute-schedule-1,f1c5e66be3575c1b,5fa19965e398e753] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:50.406 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:50.409 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:16:50.415 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:16:51.117 INFO 2981023 --- [ocp-async-4,6a0f4916c1bd2429,4e66a9811cd351f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:51.357 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fe8271684fdf20d2,8d4ce0dfb56965d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=fe8271684fdf20d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:51.367 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fe8271684fdf20d2,8d4ce0dfb56965d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=fe8271684fdf20d2, duration=13 ms] 2024-09-11 17:16:51.483 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:16:51.484 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:51.585 INFO 2981023 --- [ocp-schedules-2,68205cccabf48dc1,2b7b28f5c414c1d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:51.639 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f72934174d5b95a6,b8583125c6709711] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=f72934174d5b95a6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:51.642 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f72934174d5b95a6,b8583125c6709711] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=f72934174d5b95a6, duration=6 ms] 2024-09-11 17:16:51.716 INFO 2981023 --- [ocp-schedules-9,24d57373fa03ef3e,0265e2848f2e6ccd] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:16:51.734 WARN 2981023 --- [ocp-schedules-9,24d57373fa03ef3e,0265e2848f2e6ccd] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:16:52.013 INFO 2981023 --- [host-check-clockdiff3,fc31e50b7280b1a6,96f3dfa91eb4bc0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046212 1 1 , error= 2024-09-11 17:16:52.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:52.593 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:52.966 INFO 2981023 --- [ocp-schedules-5,7da632c4b1fb2702,7b6c7f95e00e738a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:16:52.966 INFO 2981023 --- [ocp-schedules-5,7da632c4b1fb2702,7b6c7f95e00e738a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:16:53.117 INFO 2981023 --- [ocp-async-3,9f1fab4fe9488399,5c6c8ab354ad461c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:53.173 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:53.183 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:53.183 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6d6e593eee1f665,5ad5e08d736255ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,b6d6e593eee1f665,5ad5e08d736255ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:53.956 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2162631831c248c,54c42d6247a9c41e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=b2162631831c248c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:53.966 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2162631831c248c,54c42d6247a9c41e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=b2162631831c248c, duration=13 ms] 2024-09-11 17:16:54.115 INFO 2981023 --- [ocp-schedules-3,d9030317d66ef784,c5b806a324296387] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:16:54.231 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8ded399ce418e402,11ca4e266fd6f6a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8ded399ce418e402, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:54.234 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8ded399ce418e402,11ca4e266fd6f6a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8ded399ce418e402, duration=5 ms] 2024-09-11 17:16:55.117 INFO 2981023 --- [ocp-async-2,4cc874407fa16fc3,fdde66d37e1981af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:55.234 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:16:55.234 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:16:55.267 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:16:55.267 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:16:55.273 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:16:55.273 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:16:55.274 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:16:55.275 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:16:55.287 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.288 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.288 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.289 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:16:55.291 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:16:55.294 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:16:55.294 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:16:55.295 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:16:55.295 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:16:55.296 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.297 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.300 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:16:55.300 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:16:55.301 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:16:55.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:55.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:16:55.622 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:16:55.628 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:16:55.628 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:16:55.881 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ae11b2f84b826e4b,c01db995d989c6ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ae11b2f84b826e4b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:55.891 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ae11b2f84b826e4b,c01db995d989c6ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ae11b2f84b826e4b, duration=12 ms] 2024-09-11 17:16:56.143 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7d92ac71a59b51e2,546375dcdbfbce5f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7d92ac71a59b51e2, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:56.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7d92ac71a59b51e2,546375dcdbfbce5f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7d92ac71a59b51e2, duration=6 ms] 2024-09-11 17:16:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6bf4b274aeca0f4,71336672210c771f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,b6bf4b274aeca0f4,71336672210c771f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:56.352747+08:00, lastProcessTime=2024-09-11T17:16:46.347806+08:00 2024-09-11 17:16:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:56.352747+08:00 2024-09-11 17:16:57.117 INFO 2981023 --- [ocp-async-4,365d5ae979266b03,a1e0196f7b47c373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:57.732 INFO 2981023 --- [ocp-async-9,70dd10c4ed597182,b13c3b3aaf06523a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:57.767 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,283b4a0d66cb397b,36574465af9e7b07] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=283b4a0d66cb397b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:57.777 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,283b4a0d66cb397b,36574465af9e7b07] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=283b4a0d66cb397b, duration=12 ms] 2024-09-11 17:16:57.991 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987} 2024-09-11 17:16:57.994 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obCluste 2024-09-11 17:16:57.994 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,007d693802ee1d8a,c483e293d7b705d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=007d693802ee1d8a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:58.037 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818037000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.037Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.042 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818042000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.042Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.042 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,007d693802ee1d8a,c483e293d7b705d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=007d693802ee1d8a, duration=19 ms] 2024-09-11 17:16:58.042 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818042000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.042Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.058 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818058000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.058Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.058 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818058000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.058Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.060 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818060000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.060Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.061 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818061000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.061Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.062 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818062000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.062Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.105 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.105 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.108 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.108 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 57 2024-09-11 17:16:58.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.113 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 228 2024-09-11 17:16:58.113 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.122 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.122 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.123 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.123 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.127 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.131 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.131 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.134 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.134 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.136 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.139 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.141 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.142 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.042, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.146 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.058, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.147 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.147 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.148 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.148 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.150 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.042, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.150 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.037, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.152 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.154 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.156 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.06, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.165 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.058, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.165 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.062, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.172 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.061, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.176 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818176000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.176Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818183000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.183Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.186 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818186000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.186Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.187 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.187 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.194 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818194000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.194Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.195 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818195000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.195Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.205 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.205 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.206 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.206 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818208000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.208Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.211 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818211000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.211Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.223 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.223 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.224 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.224 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.225 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.225 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.226 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.226 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.228 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.228 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.230 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.231 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.231 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.176, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.244 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.186, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.245 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.183, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.249 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.249 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.251 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.194, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.251 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.254 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.255 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.255 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.208, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.257 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.211, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.260 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.195, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.260 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.272 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:16:58.285 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818285000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.285Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:16:58.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.292 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.292 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.298 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.298 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:16:58.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.285, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.299 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.303 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.317 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.319 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:16:58.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:58.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:59.117 INFO 2981023 --- [ocp-async-7,8bd334a9a8efd27c,7e422ce890283615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,d4d04353bf1021fa,e0e58b2286443dfa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,d4d04353bf1021fa,e0e58b2286443dfa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:59.663 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1175129c0a51688c,d3928665ea48dde0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=1175129c0a51688c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:59.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1175129c0a51688c,d3928665ea48dde0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=1175129c0a51688c, duration=13 ms] 2024-09-11 17:16:59.931 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,73fb8dc43839c37c,3ae463fb5260eb88] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=73fb8dc43839c37c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:59.933 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,73fb8dc43839c37c,3ae463fb5260eb88] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=73fb8dc43839c37c, duration=5 ms] 2024-09-11 17:17:00.471 INFO 2981023 --- [ocp-schedules-19,1adac90f75234eca,303140d73babf562] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:00.506 WARN 2981023 --- [scanner-13452972-1,1adac90f75234eca,a160c0ca67b0b106] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:01.117 INFO 2981023 --- [ocp-async-10,9b38f4c1854fa14c,c54d89a0e6fc09ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:01.588 INFO 2981023 --- [ocp-schedules-13,80395de33e5f9020,a0305bd51e5c7f9d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:01.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:01.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,d467b9caf89901a0,4b5fa0e08d6c8b6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,d467b9caf89901a0,4b5fa0e08d6c8b6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:02.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99e8c6a008def68e,6ccb818f602d2c22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=99e8c6a008def68e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:02.367 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99e8c6a008def68e,6ccb818f602d2c22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=99e8c6a008def68e, duration=27 ms] 2024-09-11 17:17:02.624 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,467cbec0531a7b16,be82fb48f18a98d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=467cbec0531a7b16, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:02.631 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,467cbec0531a7b16,be82fb48f18a98d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=467cbec0531a7b16, duration=11 ms] 2024-09-11 17:17:03.118 INFO 2981023 --- [ocp-async-1,6e6dfe1ab1e6c9ef,edbd306b8775e4fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:04.279 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aad1086294ca5b54,8af013d03aab0111] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=aad1086294ca5b54, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:04.289 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aad1086294ca5b54,8af013d03aab0111] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=aad1086294ca5b54, duration=13 ms] 2024-09-11 17:17:04.588 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,cfcb7b271a772ca7,3b8efbc0e2148021] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=cfcb7b271a772ca7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:04.591 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,cfcb7b271a772ca7,3b8efbc0e2148021] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=cfcb7b271a772ca7, duration=6 ms] 2024-09-11 17:17:04.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:04.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:05.118 INFO 2981023 --- [ocp-async-8,93b7520f5f340e3a,d799df9d12abe7e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ab0a6e8f2d48dca,2b9dab0f2bcdf632] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,7ab0a6e8f2d48dca,2b9dab0f2bcdf632] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:06.352434+08:00, lastProcessTime=2024-09-11T17:16:56.352747+08:00 2024-09-11 17:17:06.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,75b363cebf9819e1,eb9284478831c1bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=75b363cebf9819e1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:06.418 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,75b363cebf9819e1,eb9284478831c1bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=75b363cebf9819e1, duration=15 ms] 2024-09-11 17:17:06.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:06.352434+08:00 2024-09-11 17:17:06.713 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b98bb1731d2908aa,6b1ae9e3a16d00c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b98bb1731d2908aa, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:06.716 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b98bb1731d2908aa,6b1ae9e3a16d00c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b98bb1731d2908aa, duration=6 ms] 2024-09-11 17:17:07.035 INFO 2981023 --- [ocp-schedules-20,b8a63ec5a2035c99,ec663e1b9702671f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:17:07.037 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:17:07.037+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:17:07.037 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:17:07.039 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:17:07.040 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:17:07.118 INFO 2981023 --- [ocp-async-10,61c21a6f3f0f8312,a671379fcbbf1f32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:07.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:07.599 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:07.732 INFO 2981023 --- [ocp-async-7,c7593e05ffb89a59,f3074435ccd7571c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,cade69d59dc33ad1,cf28443f5bbf950a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,cade69d59dc33ad1,cf28443f5bbf950a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:08.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,63910a9921c951e2,69df9fe9ecbeea02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=63910a9921c951e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:08.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,63910a9921c951e2,69df9fe9ecbeea02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=63910a9921c951e2, duration=33 ms] 2024-09-11 17:17:08.580 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b8322757ea16c7ae,89bf7c7f2c9228e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b8322757ea16c7ae, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:08.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b8322757ea16c7ae,89bf7c7f2c9228e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b8322757ea16c7ae, duration=9 ms] 2024-09-11 17:17:09.118 INFO 2981023 --- [ocp-async-1,8a3a68b2a6751454,9667fe088cef936b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:17:10.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:17:10.599 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:10.600 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:10.662 INFO 2981023 --- [ocp-schedules-3,66dd1bf89458cb0e,965477a8df432d5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:10.690 WARN 2981023 --- [scanner-13453195-1,66dd1bf89458cb0e,785d3043345724f8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:10.747 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.754 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.758 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.777 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.780 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.783 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.787 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.793 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.793 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.900 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.903 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.908 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.926 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:17:10.926 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:17:11.118 INFO 2981023 --- [ocp-async-8,da6bde86881fdb95,2885804db4914453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,08a9855683c975b9,959015304d89a2de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,08a9855683c975b9,959015304d89a2de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:11.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7577cfe1f4aea94f,9a216dae1d44712c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7577cfe1f4aea94f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:11.391 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7577cfe1f4aea94f,9a216dae1d44712c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7577cfe1f4aea94f, duration=13 ms] 2024-09-11 17:17:11.591 INFO 2981023 --- [ocp-schedules-22,678ea62bb00a5b6e,f14c0e08ccbec206] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:11.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a82eb491380553cd,4260ee8a3e26e53e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=a82eb491380553cd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:11.657 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a82eb491380553cd,4260ee8a3e26e53e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=a82eb491380553cd, duration=7 ms] 2024-09-11 17:17:12.082 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046160, nodeCount=401, affectRows=401 2024-09-11 17:17:12.207 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:17:12.225 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:12.228 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:12.303 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:12.304 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:12.320 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:12.320 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:12.333 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:12.334 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:12.346 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:17:13.118 INFO 2981023 --- [ocp-async-6,6a06383b95e9346c,effc77ba5f4f49db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:13.186 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:13.194 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:13.194 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:13.600 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:13.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:13.659 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f389073fc35e80,2caad21f0331941f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=05f389073fc35e80, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:13.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f389073fc35e80,2caad21f0331941f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=05f389073fc35e80, duration=13 ms] 2024-09-11 17:17:13.905 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=51, affectRows=51 2024-09-11 17:17:13.905 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=49, validCount=49 2024-09-11 17:17:13.942 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,84aec8587d66d9c9,cb6693859b523d4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=84aec8587d66d9c9, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:13.945 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,84aec8587d66d9c9,cb6693859b523d4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=84aec8587d66d9c9, duration=6 ms] 2024-09-11 17:17:14.044 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=1000, affectRows=1000 2024-09-11 17:17:14.045 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=1000, affectRows=1000 2024-09-11 17:17:14.047 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=1000, validCount=1000 2024-09-11 17:17:14.047 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=999, validCount=999 2024-09-11 17:17:14.112 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046160, nodeCount=401, validCount=401 2024-09-11 17:17:14.130 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046160, nodeCount=79, affectRows=79 2024-09-11 17:17:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,857e378482c5c87e,aa5661223de4635b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,675e4a6be4dbdccd,fb457a83449cf23f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:17:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,857e378482c5c87e,aa5661223de4635b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:14.443 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046160, nodeCount=79, validCount=79 2024-09-11 17:17:15.118 INFO 2981023 --- [ocp-async-4,5c1b00403ffa0fd0,15cc9dc0ae52a2cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:16.020 INFO 2981023 --- [ocp-schedules-20,81f60903753e0e98,3a8b838c38b77f13] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.021 INFO 2981023 --- [host-check-clockdiff1,81f60903753e0e98,ba226117c4ae5366] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:17:16.021 INFO 2981023 --- [ocp-schedules-7,3194b8fd1e664d04,0659e50fac637a9f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.022 INFO 2981023 --- [host-check-clockdiff3,81f60903753e0e98,38b30efe0bd21175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:17:16.022 INFO 2981023 --- [host-check-clockdiff4,81f60903753e0e98,51016309c92fd6d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:17:16.023 INFO 2981023 --- [host-check-clockdiff2,81f60903753e0e98,21f6ffaf178f9764] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:17:16.025 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:16.025 INFO 2981023 --- [ocp-schedules-9,b8e9079e82222c81,7fc408979742b901] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.028 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:16.028 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:16.029 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:16.029 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:16.044 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:16.044 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-clockdiff4,81f60903753e0e98,51016309c92fd6d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046236 0 0 , error= 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-clockdiff2,81f60903753e0e98,21f6ffaf178f9764] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046236 0 0 , error= 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:16.063 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.067 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.070 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.073 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.073 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.074 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.075 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.075 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.077 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.078 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.078 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.079 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.084 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.085 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.086 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.090 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:17:16.092 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:17:16.092863307+08:00 2024-09-11 17:17:16.094 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:17:16.093+08:00 2024-09-11 17:17:16.095 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:17:16.093+08:00 and status AVAILABLE 2024-09-11 17:17:16.101 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.109 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:17:16.124 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:17:16.126 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:17:16.125664444+08:00 2024-09-11 17:17:16.138 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:17:16.126+08:00 2024-09-11 17:17:16.138 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.170 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:17:16.126+08:00 and status AVAILABLE 2024-09-11 17:17:16.175 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:17:16.175 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:17:16.175589881+08:00 2024-09-11 17:17:16.176 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:17:16.175+08:00 2024-09-11 17:17:16.179 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:17:16.175+08:00 and status AVAILABLE 2024-09-11 17:17:16.184 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:17:16.184 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.186 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:17:16.186491274+08:00 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:17:16.191 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-28,1739e27b161aba3b,66e1c19582f0d18a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:17:16.195 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:17:16.186+08:00 2024-09-11 17:17:16.197 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.197 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.202 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.202 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:17:16.203 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.204 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.206 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:17:16.186+08:00 and status AVAILABLE 2024-09-11 17:17:16.206 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.212 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.213 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.214 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.220 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.222 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:17:16.239 INFO 2981023 --- [ocp-schedules-29,1aa0956fdebed3bf,1e136f378ef64007] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:17:16.242 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.247 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.254 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.256 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.257 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.267 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.274 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:17:16.280 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:17:16.287 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.293 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.294 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.294 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:17:16.302 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:17:16.302 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:17:16.327 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:16.328 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:17:16.339 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.343 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.343 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.344 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:16.349 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.350 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.353 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.353 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.354 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:17:16.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.357 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:17:16.371 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:17:16.371 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:17:16.371 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.372 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.373 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:17:16.381 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.382 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:16.382 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:16.381779+08:00, lastProcessTime=2024-09-11T17:17:06.352434+08:00 2024-09-11 17:17:16.388 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:17:16.389 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:17:16.397 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.406 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:17:16.413 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.422 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:17:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.425 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.426 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:16.427 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:17:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.429 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:16.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:16.434 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:17:16.439 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:17:16.450 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:17:16.454 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:16.458 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:17:16.467 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:17:16.479 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.480 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:17:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.491 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:17:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:17:16.493 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.503 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.507 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:17:16.518 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:17:16.520 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:17:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:16.527 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:17:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.539 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:16.544 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:16.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:17:16.576 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:17:16.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.581 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.593 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:17:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:16.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:16.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:17:16.626 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:17:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:16.636 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:16.381779+08:00 2024-09-11 17:17:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:17:16.641 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:17.118 INFO 2981023 --- [ocp-async-3,308e03bf103a6360,e026f155850d3056] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,5bb503ffdcaf98fb,058a79706bc1c08e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,5bb503ffdcaf98fb,058a79706bc1c08e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:17.732 INFO 2981023 --- [ocp-async-1,10b900b5aa8c4991,7226cb5faf1a3c3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:17.814 INFO 2981023 --- [host-check-clockdiff1,81f60903753e0e98,ba226117c4ae5366] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046237 0 0 , error= 2024-09-11 17:17:18.562 INFO 2981023 --- [host-check-clockdiff3,81f60903753e0e98,38b30efe0bd21175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046238 1 1 , error= 2024-09-11 17:17:19.118 INFO 2981023 --- [ocp-async-10,99db165f6c879ab7,345b953e6ff37092] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:19.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:19.602 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,f74cebbb6b35a1d1,49c6ab034e9f19ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,f74cebbb6b35a1d1,49c6ab034e9f19ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:20.968 INFO 2981023 --- [ocp-schedules-30,6703530af43bac2e,6a7b609d880175a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:20.983 WARN 2981023 --- [scanner-13455029-1,6703530af43bac2e,170f8e39d0bbee3e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:21.119 INFO 2981023 --- [ocp-async-4,5b20b1809514a6d9,65575de0a0508153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:21.594 INFO 2981023 --- [ocp-schedules-9,eef3b71a6f30d850,3fa81643f9248e08] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:21.997 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:21.997 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:17:22.001 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:22.002 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:22.002 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:22.002 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:22.003 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:22.020 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.020 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.025 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:17:22.602 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:22.603 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:23.119 INFO 2981023 --- [ocp-async-3,41e68bc67ed95365,0765172904f42d0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,6fcc42d47260d144,cee28043c5a92a49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,6fcc42d47260d144,cee28043c5a92a49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:24.118 INFO 2981023 --- [ocp-schedules-8,7ecf35265b5775b8,71d1fe7d33fac2b0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:17:25.119 INFO 2981023 --- [ocp-async-7,27a6cd559c6ab4fc,09ab56aff5370e13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:25.603 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:25.604 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2b2bb08275b0cea7,d78beef64dc28e89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2b2bb08275b0cea7,d78beef64dc28e89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:26.336983+08:00, lastProcessTime=2024-09-11T17:17:16.381779+08:00 2024-09-11 17:17:26.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:26.336983+08:00 2024-09-11 17:17:27.119 INFO 2981023 --- [ocp-async-2,bbc3e052399d04e3,54a02eb54fb7962a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:27.732 INFO 2981023 --- [ocp-async-10,01b4fc6256fb7adb,3c912c93f97f23c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:28.604 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:28.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:29.119 INFO 2981023 --- [ocp-async-5,5c82fdfd392be87a,f8b00c8579f21297] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9c0b9b7f74726d1b,10e2b8c22e6cac36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,9c0b9b7f74726d1b,10e2b8c22e6cac36] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:31.119 INFO 2981023 --- [ocp-async-6,1b4273a9f03bd975,fc8a12fbccb3a120] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:31.153 INFO 2981023 --- [ocp-schedules-14,4118350254a53b73,a298453cb4c692af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:31.297 WARN 2981023 --- [scanner-13455218-1,4118350254a53b73,264ba6c51695cb31] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:31.330 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.334 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.338 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.352 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.352 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.356 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.356 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.360 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.362 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.488 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.494 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.499 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.597 INFO 2981023 --- [ocp-schedules-16,264f629d3713f0d0,883c46038de1467c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:31.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:31.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,21d737339da66273,dfd9362ed093e377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,21d737339da66273,dfd9362ed093e377] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:33.119 INFO 2981023 --- [ocp-async-9,cb22c4c36395b439,6c4081dd13fee4c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:33.197 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:33.205 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:33.205 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:34.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:34.607 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:35.119 INFO 2981023 --- [ocp-async-3,9ba163d91beb5b98,1d75da8a2cac9b44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,30ba455be8bdfc1a,e55d5df915bd855b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,30ba455be8bdfc1a,e55d5df915bd855b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:36.165 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7e7407bac282880a,78aba385f6527084] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7e7407bac282880a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:36.187 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7e7407bac282880a,78aba385f6527084] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7e7407bac282880a, duration=26 ms] 2024-09-11 17:17:36.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:36.351389+08:00, lastProcessTime=2024-09-11T17:17:26.336983+08:00 2024-09-11 17:17:36.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:36.351389+08:00 2024-09-11 17:17:36.568 INFO 2981023 --- [ocp-schedules-1,a6bebb5d1cf49350,df7d2f990aa7c350] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:17:36.585 WARN 2981023 --- [ocp-schedules-1,a6bebb5d1cf49350,df7d2f990aa7c350] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:17:37.119 INFO 2981023 --- [ocp-async-7,aca5bfd6932b8d60,09036cdf630c880e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:37.314 INFO 2981023 --- [prometheus-cache-monitor1,41ad6dbe6bae398c,70852fc49bc076d9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:17:37.315 INFO 2981023 --- [prometheus-cache-monitor1,41ad6dbe6bae398c,70852fc49bc076d9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:17:37.607 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:37.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:37.732 INFO 2981023 --- [ocp-async-5,87ab6aed07cae8a4,777c79331cc80591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:37.873 INFO 2981023 --- [ocp-schedules-25,2acf5ab31b7fa168,5b5d58fb5ae1c81b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:17:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,ac0e9c76b897f4f9,4cff8b90fa8e138a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:38.315 INFO 2981023 --- [prometheus-minute-schedule-1,ac0e9c76b897f4f9,4cff8b90fa8e138a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:39.119 INFO 2981023 --- [ocp-async-2,83d4e745e040e079,56667f8758ec1ebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:40.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:17:40.004 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:17:40.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:40.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:41.120 INFO 2981023 --- [ocp-async-4,a0d91c89a02114a6,f652f9b2329fcdee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,30728fa6dfb59c02,cd6396b9fac965d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,30728fa6dfb59c02,cd6396b9fac965d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:41.532 INFO 2981023 --- [ocp-schedules-4,3c31328c3704768e,1a76f90be1031a9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:41.572 WARN 2981023 --- [scanner-13455559-1,3c31328c3704768e,c1151fe745055fde] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:41.599 INFO 2981023 --- [ocp-schedules-14,9c30c19b706cbc19,115a854b14499094] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:43.120 INFO 2981023 --- [ocp-async-5,57f38c11191503f4,b9e36457fa6534c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:43.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:43.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,3798cc2ccfce1062,edfe862c0bd17cc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,3798cc2ccfce1062,edfe862c0bd17cc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:45.101 INFO 2981023 --- [prometheus-second-schedule-1,fbbe531bc2cd626e,53e739d19e8f3331] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:17:45.120 INFO 2981023 --- [ocp-async-10,6e52a05ff0a04588,8570815192544a82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:46.020 INFO 2981023 --- [ocp-schedules-2,1ccad5e156179f1f,8ce2b54e3a1095c2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.020 INFO 2981023 --- [host-check-clockdiff1,1ccad5e156179f1f,84f501aa0028d35e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:17:46.021 INFO 2981023 --- [host-check-clockdiff2,1ccad5e156179f1f,5c67e56ee1a994e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:17:46.021 INFO 2981023 --- [host-check-clockdiff3,1ccad5e156179f1f,6b8d02504c1674b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:17:46.022 INFO 2981023 --- [host-check-clockdiff4,1ccad5e156179f1f,103a218ebafbf298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:17:46.026 INFO 2981023 --- [ocp-schedules-18,c16404f3aefd5ece,e0e5d75c86373176] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.031 INFO 2981023 --- [ocp-schedules-27,f279e4160a89dfce,54281717a1d36f42] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:46.033 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:46.037 INFO 2981023 --- [host-check-clockdiff4,1ccad5e156179f1f,103a218ebafbf298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046266 0 0 , error= 2024-09-11 17:17:46.038 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:46.041 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.042 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.042 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.043 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.044 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.046 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.046 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.047 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.047 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.049 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.049 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.051 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.089 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.089 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.090 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.090 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:17:46.092 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:17:46.091875122+08:00 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.094 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:17:46.094 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:17:46.096143242+08:00 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:17:46.096+08:00 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:17:46.097046845+08:00 2024-09-11 17:17:46.097 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:17:46.096+08:00 2024-09-11 17:17:46.098 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:17:46.096+08:00 and status AVAILABLE 2024-09-11 17:17:46.099 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:17:46.096+08:00 and status AVAILABLE 2024-09-11 17:17:46.102 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:17:46.092+08:00 2024-09-11 17:17:46.104 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:17:46.092+08:00 and status AVAILABLE 2024-09-11 17:17:46.105 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:17:46.105 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.117 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:17:46.117 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:17:46.117729124+08:00 2024-09-11 17:17:46.118 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:17:46.117+08:00 2024-09-11 17:17:46.120 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:17:46.117+08:00 and status AVAILABLE 2024-09-11 17:17:46.124 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:17:46.131 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.132 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:17:46.132 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:17:46.133 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.134 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.136 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.145 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.147 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.148 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.150 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.162 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.164 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:17:46.165 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.169 INFO 2981023 --- [ocp-schedules-28,6b680598c89dc853,ed3ad5f9399df552] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:17:46.177 INFO 2981023 --- [ocp-schedules-28,6b680598c89dc853,ed3ad5f9399df552] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:17:46.177 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-10,5ba648f694f94b20,62b178d2fdc6a3b6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:17:46.191 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:17:46.198 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.200 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.202 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.202 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:17:46.217 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:17:46.228 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:46.232 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.243 INFO 2981023 --- [ocp-schedules-3,edbf5d9cee14e7e8,7f72e4007f56234a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:17:46.256 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:17:46.285 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.288 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.291 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.292 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.293 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:17:46.294 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.295 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.295 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.296 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:17:46.297 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.309 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.309 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:17:46.310 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.310 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.311 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.311 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.316 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:17:46.319 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:17:46.324 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:17:46.331 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:17:46.338 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:17:46.344 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:17:46.344 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:46.342869+08:00, lastProcessTime=2024-09-11T17:17:36.351389+08:00 2024-09-11 17:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.349 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.356 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:17:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.364 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.365 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.376 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.380 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.393 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.394 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.394 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.394 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.395 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.395 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.398 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.401 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:17:46.409 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:17:46.411 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:17:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:17:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:46.418 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:17:46.419 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:17:46.424 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:17:46.430 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:17:46.432 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:17:46.454 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.459 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:17:46.484 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:17:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.489 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:46.495 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:46.499 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:17:46.499 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:17:46.509 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.509 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.510 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:17:46.510 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:17:46.511 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.511 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.512 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.512 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:46.518 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:17:46.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.536 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:46.542 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.543 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.546 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:17:46.548 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:17:46.552 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:17:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.569 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:17:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.575 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:46.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.576 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.577 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.577 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.578 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.578 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:46.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:46.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:46.581 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:17:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:46.594 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:17:46.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:46.610 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:17:46.619 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:17:46.624 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:17:46.625 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.649 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.649 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.651 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:17:46.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:46.691 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:46.342869+08:00 2024-09-11 17:17:46.697 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:17:46.699 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:17:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:17:46.709 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.715 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:17:46.720 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:17:46.723 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:17:46.834 INFO 2981023 --- [ocp-schedules-17,eca42ba3fd099579,d08644c954f09125] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:17:46.834 INFO 2981023 --- [ocp-schedules-17,eca42ba3fd099579,d08644c954f09125] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:17:46.967 INFO 2981023 --- [host-check-clockdiff2,1ccad5e156179f1f,5c67e56ee1a994e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046266 0 0 , error= 2024-09-11 17:17:47.120 INFO 2981023 --- [ocp-async-3,f1f1446916ab1d32,814d13435f934bc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,f72348386c4826c1,fe43774f3580293c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,f72348386c4826c1,fe43774f3580293c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:47.733 INFO 2981023 --- [ocp-async-1,cf588ebf9bbc2575,eceee7618cdfbd4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:49.120 INFO 2981023 --- [ocp-async-5,785fb426fa16d782,9c934a6b9ad352e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:49.610 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:49.612 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:49.718 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:17:49.718+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:17:49.721 INFO 2981023 --- [host-check-clockdiff1,1ccad5e156179f1f,84f501aa0028d35e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046269 0 0 , error= 2024-09-11 17:17:49.750 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:17:49.898 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 17:17:49.900 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:17:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:17:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:17:50.171 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.180 INFO 2981023 --- [ocp-schedules-3,11920b0116e0cb1d,95a2f6b5d2ef090f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:17:50.233 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:17:50.235 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.270 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.271 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.273 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.285 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.288 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.290 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.290 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.291 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.292 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.299 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1591D753-0-0] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.303 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.304 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.312 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.312 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,0474537e59577610,e60e3104a712ed07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.314 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.314 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.315 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.315 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:17:50.317 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,0474537e59577610,e60e3104a712ed07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:50.318 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.327 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.328 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.329 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:17:50.330 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1591D75E-0-0] 2024-09-11 17:17:50.330 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:17:50.336 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:17:50.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.356 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:17:50.366 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.368 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:50.386 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:17:51.120 INFO 2981023 --- [ocp-async-2,be7caa5e04cf2096,7b081aee2f119c2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:51.485 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:17:51.486 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.487 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:51.605 INFO 2981023 --- [ocp-schedules-3,4c609d679c02d162,91cbea0dc6adc06b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:51.723 INFO 2981023 --- [ocp-schedules-14,c7ade940b2150d38,d49976fac83c7fa2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:51.765 INFO 2981023 --- [ocp-schedules-11,8ecd85690c8f8280,6fbd2aee34ae8b36] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:17:51.798 WARN 2981023 --- [ocp-schedules-11,8ecd85690c8f8280,6fbd2aee34ae8b36] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:17:51.805 WARN 2981023 --- [scanner-13455803-1,c7ade940b2150d38,be9e46b8130478d9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:51.836 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:51.841 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:17:51.847 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:17:51.848 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:51.851 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:17:51.853 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:51.857 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:17:51.866 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:17:51.870 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:17:51.993 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:52.001 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:17:52.014 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:17:52.032 INFO 2981023 --- [host-check-clockdiff3,1ccad5e156179f1f,6b8d02504c1674b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046272 1 1 , error= 2024-09-11 17:17:52.612 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:52.614 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:52.973 INFO 2981023 --- [ocp-schedules-22,af1dad9c566fbee7,9162b480d3ccab9b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:17:52.973 INFO 2981023 --- [ocp-schedules-22,af1dad9c566fbee7,9162b480d3ccab9b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:17:53.120 INFO 2981023 --- [ocp-async-1,ad49f1404de663cc,178c7b44921eb038] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:53.208 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:53.216 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:53.216 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,c7f08d1674612198,fe75f9e2c2c690cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,c7f08d1674612198,fe75f9e2c2c690cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:54.121 INFO 2981023 --- [ocp-schedules-26,fd4c173664afa72c,3f9800be5bc1b90a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:17:55.120 INFO 2981023 --- [ocp-async-8,2fff02c28114517d,e1211b87a465db0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:55.301 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:17:55.301 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:17:55.333 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:17:55.335 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:17:55.335 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:17:55.338 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:17:55.341 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:17:55.342 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:17:55.359 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.359 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.360 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:17:55.360 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.362 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:17:55.362 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:17:55.363 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:17:55.365 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.365 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:17:55.367 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:17:55.367 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:17:55.368 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:17:55.596 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:17:55.596 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:17:55.614 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:55.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:55.628 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:17:55.634 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:17:55.634 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:17:55.745 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ec65929c4a66107e,71b7973f94cce0ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ec65929c4a66107e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:55.769 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ec65929c4a66107e,71b7973f94cce0ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ec65929c4a66107e, duration=29 ms] 2024-09-11 17:17:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,fb2d9ebfc20fced2,30d1effd426cfe7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,fb2d9ebfc20fced2,30d1effd426cfe7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:56.351700+08:00, lastProcessTime=2024-09-11T17:17:46.342869+08:00 2024-09-11 17:17:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:56.351700+08:00 2024-09-11 17:17:57.120 INFO 2981023 --- [ocp-async-2,4b4013a630a6ef0b,944e23edd167f0fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:57.733 INFO 2981023 --- [ocp-async-9,96d22fdb3937ca5f,a4656f16e4cf6ab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:58.009 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.012 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obC 2024-09-11 17:17:58.015 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.018 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.020 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.022 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.026 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.029 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.032 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.035 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.037 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.038 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.038 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.039 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.049 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.053 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.057 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.061 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.062 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.063 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.064 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.064 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.065 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.066 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:17:58.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:58.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:59.121 INFO 2981023 --- [ocp-async-1,165a2d09f25b945d,cf5c35fde174232d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,599216fca57684e7,3303d34e6c976ca6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,599216fca57684e7,3303d34e6c976ca6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:01.125 INFO 2981023 --- [ocp-async-5,2bd2e59a96bc54af,695a8002ff7b18b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:01.608 INFO 2981023 --- [ocp-schedules-8,088efcd09884418a,d2ece8543af8cf12] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:01.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:01.616 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:01.927 INFO 2981023 --- [ocp-schedules-27,2fbb998ca7aec2be,4a8163dda62d2973] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:01.927 INFO 2981023 --- [ocp-schedules-27,2fbb998ca7aec2be,4a8163dda62d2973] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:18:02.048 INFO 2981023 --- [ocp-schedules-16,9fc2cd38d89e8136,472ad1785aeaa3fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:02.051 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:18:02.054 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:02.095 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:02.095 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:18:02.101 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:02.105 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:18:02.106 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000} 2024-09-11 17:18:02.108 WARN 2981023 --- [scanner-13456110-1,9fc2cd38d89e8136,39530c56b2f9b56a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:02.114 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:18:02.123 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000222, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:18:02.156 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:18:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ec184fb5165277d,45e412fbe08f13a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ec184fb5165277d,45e412fbe08f13a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:02.907 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 17:18:02.907 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 17:18:02.907 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 17:18:02.913 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000222, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:18:02.913 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 17:18:03.121 INFO 2981023 --- [ocp-async-2,1d7729124fc634af,272c5c7d449bfe14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:03.686 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000222 2024-09-11 17:18:04.616 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:04.617 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:05.121 INFO 2981023 --- [ocp-async-7,f09c5778ccd14936,e5a76ef1c951044e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f9f8f5dde61d0559,0d53612ae18be74d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,f9f8f5dde61d0559,0d53612ae18be74d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:18:05.981 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:18:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:18:05.991 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 17:18:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:18:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:18:05.994 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:05.994 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:18:05.995 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:05.995 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:18:06.044 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:15:41, start=2024-09-11T17:12:41, end=2024-09-11T17:15:11, updates=284, granule=120000ms, delay=180000ms 2024-09-11 17:18:06.051 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:15:41, target=2024-09-11T17:15:11, granule=120000ms, delay=30000ms) 2024-09-11 17:18:06.155 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:15:11, start=2024-09-11T17:00:51, end=2024-09-11T17:10:51, updates=387, granule=600000ms, delay=860000ms 2024-09-11 17:18:06.160 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:15:11, target=2024-09-11T17:10:51, granule=600000ms, delay=260000ms) 2024-09-11 17:18:06.160 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=193ms, period=120000ms 2024-09-11 17:18:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:06.337390+08:00, lastProcessTime=2024-09-11T17:17:56.351700+08:00 2024-09-11 17:18:06.468 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:06.337390+08:00 2024-09-11 17:18:07.051 INFO 2981023 --- [ocp-schedules-4,dd463a4516c53d12,a71837149ba3fd80] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:18:07.055 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:18:07.055+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:18:07.056 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:18:07.057 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:18:07.057 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:18:07.121 INFO 2981023 --- [ocp-async-6,a82a4d6f6c79ee88,0e86fe82e0709851] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:07.617 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:07.618 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:07.733 INFO 2981023 --- [ocp-async-1,7936a91297d2b5cb,8d19f1d3a485eeb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,75dcb84542e1490e,eaeb519adb2e1bea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,75dcb84542e1490e,eaeb519adb2e1bea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:09.121 INFO 2981023 --- [ocp-async-10,ae79f1533fbb7074,8dae044b2be3c907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:18:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:18:10.618 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:10.619 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:18:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:18:11.121 INFO 2981023 --- [ocp-async-7,c053708ce3203238,4f6f4326981375d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bf430de7abbd0b9,cc7e5fe3c46d65ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:11.320 INFO 2981023 --- [prometheus-minute-schedule-1,6bf430de7abbd0b9,cc7e5fe3c46d65ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:11.611 INFO 2981023 --- [ocp-schedules-30,305ce07f2165dc8c,fa80d2be45ceac59] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:12.287 INFO 2981023 --- [ocp-schedules-23,2588921b27719736,22beeae0622dd389] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:12.336 WARN 2981023 --- [scanner-13457785-1,2588921b27719736,42582f5860c7520b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:12.366 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.370 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.379 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.402 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.406 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.409 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.410 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.414 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.419 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.541 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:12.547 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:12.550 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:13.121 INFO 2981023 --- [ocp-async-9,7c7e63898d65d44c,743d4ee23f88760b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:13.219 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:18:13.230 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:13.230 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:13.619 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:13.620 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:14.006 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=1000, affectRows=1000 2024-09-11 17:18:14.008 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=999, validCount=999 2024-09-11 17:18:14.055 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=21, affectRows=21 2024-09-11 17:18:14.055 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=21, validCount=21 2024-09-11 17:18:14.058 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=27, affectRows=27 2024-09-11 17:18:14.058 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=25, validCount=25 2024-09-11 17:18:14.087 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=1000, affectRows=1000 2024-09-11 17:18:14.090 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=1000, validCount=1000 2024-09-11 17:18:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c941a60aff938a50,878852697f3e50cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,191aaa14a4522df3,bad9f94e935d7529] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:18:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c941a60aff938a50,878852697f3e50cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:14.546 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046220, nodeCount=480, affectRows=480 2024-09-11 17:18:15.122 INFO 2981023 --- [ocp-async-5,b62068295eab72c3,dd6aae2060f3ce6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:16.020 INFO 2981023 --- [ocp-schedules-16,2678d6748d520ae2,34108a77642471db] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.021 INFO 2981023 --- [ocp-schedules-25,f1ac4286ce7cd150,1a917601b8de24b5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.021 INFO 2981023 --- [host-check-clockdiff2,2678d6748d520ae2,eece4053cac91780] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:18:16.021 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:16.022 INFO 2981023 --- [host-check-clockdiff3,2678d6748d520ae2,4378e6534f5316ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:18:16.022 INFO 2981023 --- [host-check-clockdiff4,2678d6748d520ae2,5084b5dd42f55a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:18:16.023 INFO 2981023 --- [host-check-clockdiff1,2678d6748d520ae2,f2bf02e28d3756e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:18:16.023 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:16.026 INFO 2981023 --- [ocp-schedules-12,d4ee40a8278dee3d,aa236d7143bdb3bf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.026 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:16.029 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:16.029 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:16.046 INFO 2981023 --- [host-check-clockdiff4,2678d6748d520ae2,5084b5dd42f55a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046296 0 0 , error= 2024-09-11 17:18:16.066 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.067 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.070 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.070 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.075 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.077 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.090 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.192 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:18:16.192 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.201 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-21,15cc8a29adc970ef,380e7a66185c8141] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-4,c77334bc8bdb2302,6cf87a95fbf6b50f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:18:16.269 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.288 INFO 2981023 --- [ocp-schedules-21,15cc8a29adc970ef,380e7a66185c8141] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.290 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.290 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:16.296 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.298 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:18:16.456 INFO 2981023 --- [ocp-schedules-26,10e281746ebbc948,c47db187ec5ed70d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:18:16.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:16.472768+08:00, lastProcessTime=2024-09-11T17:18:06.337390+08:00 2024-09-11 17:18:16.475 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.480 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.482 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.483 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:18:16.485 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:18:16.484160726+08:00 2024-09-11 17:18:16.485 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:18:16.485063365+08:00 2024-09-11 17:18:16.488 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:18:16.488 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:18:16.485+08:00 2024-09-11 17:18:16.489 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:18:16.490 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.490 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:18:16.485+08:00 and status AVAILABLE 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:18:16.490803532+08:00 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:18:16.490966705+08:00 2024-09-11 17:18:16.491 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:18:16.485+08:00 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.493 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.494 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:18:16.485+08:00 and status AVAILABLE 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:18:16.491+08:00 2024-09-11 17:18:16.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.498 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:18:16.491+08:00 2024-09-11 17:18:16.499 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:18:16.491+08:00 and status AVAILABLE 2024-09-11 17:18:16.500 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:18:16.491+08:00 and status AVAILABLE 2024-09-11 17:18:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.501 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.502 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:18:16.503 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:16.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.508 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:18:16.512 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:18:16.513 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:18:16.513 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.571 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.571 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.573 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.574 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.575 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046220, nodeCount=480, validCount=480 2024-09-11 17:18:16.578 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.584 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.594 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:18:16.615 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.615 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.620 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:16.621 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:16.622 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.630 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:18:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.641 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:18:16.641 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.642 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.642 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:18:16.643 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.644 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:18:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:18:16.646 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:18:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:18:16.660 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:18:16.661 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:18:16.664 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:18:16.667 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:18:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:18:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:18:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.698 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.699 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.699 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.700 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.717 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value f