2023-05-15 13:53:32.074 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.081 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:53:32.086 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.086 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.086 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.087 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:53:32.089 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:53:32.090 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:53:32.090 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:53:32.090 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.091 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:53:32.092 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:53:32.092 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.092 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.092 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.092 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:53:32.093 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:53:32.096 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.097 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.097 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:53:32.100 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:53:32.101 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:53:32.101 INFO 11 --- [ForkJoinPool-65954-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129536321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:53:32.117 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.117 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:53:32.133 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.133 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:53:32.136 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116157, status:SUCCESS 2023-05-15 13:53:32.137 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116157, status:SUCCESS 2023-05-15 13:53:32.144 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.144 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:53:32.154 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.154 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:53:32.163 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.163 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:53:32.174 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:53:32.188 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:53:32.189 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.189 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.189 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.190 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:53:32.194 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:53:32.194 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:53:32.195 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:53:32.214 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:53:32.214 INFO 11 --- [ForkJoinPool-65954-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129546903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:53:32.413 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:32.414 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:32.769 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d0161526be29416e,febea714b5ff] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=d0161526be29416e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:53:32.771 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d0161526be29416e,febea714b5ff] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=d0161526be29416e, duration=3 ms] 2023-05-15 13:53:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:53:34.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:53:34.828501+08:00, lastProcessTime=2023-05-15T13:53:24.831935+08:00 2023-05-15 13:53:34.871 INFO 11 --- [alarm-detect-12,,762f035788b2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684129994 ~ 1684130004 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.27, values={1684130000=245.27}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:53:34.905 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:53:34.828501+08:00 2023-05-15 13:53:35.414 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:35.415 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:35.741 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:35.744 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:35.745 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:35.763 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:36.592 INFO 11 --- [pool-ocp-schedules-18,,e48d83ce3c7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:53:36.725 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:38.247 INFO 11 --- [pool-overview-container-7,,83d56293a67e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:53:38.415 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:38.416 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:38.498 INFO 11 --- [pool-scanner--15965114-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:53:39.111 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c4049ad8c54b456d,28f243a2bd6e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=c4049ad8c54b456d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:53:39.116 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c4049ad8c54b456d,28f243a2bd6e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=c4049ad8c54b456d, duration=5 ms] 2023-05-15 13:53:39.623 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,541cbb0886bc4438,18872fdc027f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=541cbb0886bc4438, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:53:39.629 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,541cbb0886bc4438,18872fdc027f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=541cbb0886bc4438, duration=6 ms] 2023-05-15 13:53:41.163 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,4f89b50e01e64a16,8da92f0b0a5c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=4f89b50e01e64a16, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:53:41.169 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,4f89b50e01e64a16,8da92f0b0a5c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=4f89b50e01e64a16, duration=6 ms] 2023-05-15 13:53:41.416 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:41.417 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:42.184 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:44.417 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:44.419 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:44.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:53:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:53:44.831540+08:00, lastProcessTime=2023-05-15T13:53:34.828501+08:00 2023-05-15 13:53:44.869 INFO 11 --- [alarm-detect-12,,7a25023c56a8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130004 ~ 1684130014 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130010=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:53:44.903 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:53:44.831540+08:00 2023-05-15 13:53:45.439 INFO 11 --- [pool-ocp-schedules-20,,ec87fbb814df] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:53:45.744 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:45.746 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:45.748 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:45.766 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:46.614 INFO 11 --- [pool-ocp-schedules-4,,4be69e0a8611] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:53:46.729 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:47.419 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:47.420 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:48.248 INFO 11 --- [pool-overview-container-1,,1697667ef36a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:53:48.501 INFO 11 --- [pool-scanner--15965214-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:53:50.421 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:50.422 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:51.567 INFO 11 --- [pool-ocp-schedules-17,,00d77585aa6a] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:53:51.567 INFO 11 --- [pool-ocp-schedules-17,,00d77585aa6a] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:53:51.746 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:51.748 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:51.752 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:53.422 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:53.423 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,3750116fe9fa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40108} 2023-05-15 13:53:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,3750116fe9fa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:53:53.880 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,14070d042ebf43ed,ca2a85334416] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=14070d042ebf43ed, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:53:53.887 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,14070d042ebf43ed,ca2a85334416] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=14070d042ebf43ed, duration=7 ms] 2023-05-15 13:53:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:53:54.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:53:54.831976+08:00, lastProcessTime=2023-05-15T13:53:44.831540+08:00 2023-05-15 13:53:54.862 INFO 11 --- [alarm-detect-7,,15c389dd9d99] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130014 ~ 1684130024 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130020=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:53:54.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:53:54.831976+08:00 2023-05-15 13:53:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,0415791c74a6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:53:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,e6333b73c8df] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:53:55.747 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:55.752 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:55.754 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:55.771 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:56.423 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:56.424 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:53:56.634 INFO 11 --- [pool-ocp-schedules-8,,aa679d669db5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:53:56.740 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:57.187 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:53:58.248 INFO 11 --- [pool-overview-container-5,,96bbbe52819a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:53:58.501 INFO 11 --- [pool-scanner--15965308-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:53:59.425 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:53:59.426 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:54:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130040 2023-05-15 13:54:00.014 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 13:54:00.014 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 13:54:00.014 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 13:54:00.014 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 13:54:00.018 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 13:54:00.019 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 13:54:00.019 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 13:54:00.019 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 13:54:00.040 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:54:00.057 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 13:54:00.065 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 13:54:00.066 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 13:54:00.066 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 13:54:00.066 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 13:54:00.066 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 13:54:00.073 WARN 11 --- [pool-ocp-schedules-6,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 13:54:00.073 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 13:54:00.599 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 13:54:00.968 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:52:00.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:00 CST 2023 2023-05-15 13:54:00.968 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 13:52:00.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 13:54:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:00.995 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:00.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:00 CST 2023 2023-05-15 13:54:00.995 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:53:00.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:00.999 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:54:00.999 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:00.999 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:01.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:00.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:01 CST 2023 2023-05-15 13:54:01.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:53:00.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:01.023 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:54:01.023 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:01.023 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:01.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:00.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:01 CST 2023 2023-05-15 13:54:01.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:53:00.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:01.045 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:54:01.045 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:01.045 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:01.050 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:30.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:01 CST 2023 2023-05-15 13:54:01.051 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:53:30.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:01.054 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:54:01.054 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:01.054 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:01.060 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:00.0, nextRun=2023-05-15 13:54:00.0, currentDate=Mon May 15 13:54:01 CST 2023 2023-05-15 13:54:01.060 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:53:00.0, nextRunTime=2023-05-15 13:54:00.0} 2023-05-15 13:54:01.063 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:54:01.064 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:01.064 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:01.080 WARN 11 --- [pool-ocp-schedules-6,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 13:54:01.081 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.062623385S 2023-05-15 13:54:01.081 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 13:54:01.204 INFO 11 --- [pool-ocp-schedules-6,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 13:54:01.281 INFO 11 --- [pool-ocp-schedules-6,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 13:54:01.333 INFO 11 --- [pool-ocp-schedules-6,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 13:54:01.343 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 13:54:01.445 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131744, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.463 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131744, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:54:01.432+08:00, endTime=null} 2023-05-15 13:54:01.478 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131745, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.481 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131745, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:54:01.475+08:00, endTime=null} 2023-05-15 13:54:01.487 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,f06f953541be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131744, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116163, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.432+08:00, sub_task_instance_id=131744}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.487 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,f06f953541be] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:54:01.496 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131739, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.498 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131739, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T13:54:01.492+08:00, endTime=null} 2023-05-15 13:54:01.504 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ea85fb9d629c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131745, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116164, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.475+08:00, sub_task_instance_id=131745}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.511 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131742, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,5e579a33bc6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130100, interval=60 2023-05-15 13:54:01.513 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131742, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:54:01.508+08:00, endTime=null} 2023-05-15 13:54:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,35f3cdb38474] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130100, interval=60 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:54:01.514 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:54:01.520 INFO 11 --- [pool-ocp-schedules-6,,] o.s.boot.SpringApplication : Started application in 1.505 seconds (JVM running for 599697.493) 2023-05-15 13:54:01.520 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.506193231S 2023-05-15 13:54:01.525 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ad4712d6667d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131739, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116159, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.492+08:00, sub_task_instance_id=131739}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.532 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.533 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.533 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.533 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ea85fb9d629c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131745, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.534 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.534 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.534 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.534 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.535 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.535 INFO 11 --- [pool-prometheus-minute-schedule-1,,5e579a33bc6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:54:01.535 INFO 11 --- [pool-prometheus-minute-schedule-1,,5e579a33bc6e] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130041, count=8, elapsed=1682445911494 2023-05-15 13:54:01.535 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.535 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.535 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.536 INFO 11 --- [pool-prometheus-minute-schedule-2,,35f3cdb38474] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:54:01.536 INFO 11 --- [pool-prometheus-minute-schedule-2,,35f3cdb38474] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130041, count=6, elapsed=1682445911495 2023-05-15 13:54:01.537 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6f6ebbe9e433] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131742, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116161, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.508+08:00, sub_task_instance_id=131742}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.537 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.538 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.538 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131743, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.539 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:01.541 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131743, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:54:01.525+08:00, endTime=null} 2023-05-15 13:54:01.542 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.542 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.543 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:54:01.543 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.543 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.543 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:01.544 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.544 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:54:01.544 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:54:01.546 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:54:01.547 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ad4712d6667d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131739, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.548 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:54:01.548 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:54:01.548 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.548 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.548 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:54:01.552 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:54:01.555 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.555 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.555 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:54:01.557 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:01.557 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:01.557 INFO 11 --- [pool-3240998-thread-2,fb27d19334114ed7,3ec7cc8b7586] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:54:01.557 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:54:01.557 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131740, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.558 INFO 11 --- [pool-3240998-thread-6,fb27d19334114ed7,29bff1caeb92] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:54:01.558 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:54:01.559 INFO 11 --- [pool-3240998-thread-4,fb27d19334114ed7,37aea2f963f0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:54:01.559 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131740, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T13:54:01.555+08:00, endTime=null} 2023-05-15 13:54:01.559 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7f4b767c6082] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131743, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116162, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.525+08:00, sub_task_instance_id=131743}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.559 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6f6ebbe9e433] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:54:01.560 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:54:01.560 INFO 11 --- [pool-3240998-thread-1,fb27d19334114ed7,0db85e9d3fe9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:54:01.560 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6f6ebbe9e433] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:54:01.562 INFO 11 --- [pool-3240998-thread-5,fb27d19334114ed7,658404e2eeec] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:54:01.564 INFO 11 --- [pool-3240998-thread-3,fb27d19334114ed7,c0e264f6da01] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:54:01.565 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6f6ebbe9e433] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131742, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.565 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,f06f953541be] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:54:01.568 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131741, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:01.572 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,f06f953541be] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131744, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.575 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131741, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:54:01.566+08:00, endTime=null} 2023-05-15 13:54:01.576 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b5555b0d90d6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131740, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116159, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.554+08:00, sub_task_instance_id=131740}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.578 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7f4b767c6082] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:54:01.578 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7f4b767c6082] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:54:01.580 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7f4b767c6082] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 13:54:01.584 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7f4b767c6082] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131743, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.586 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b5555b0d90d6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131740, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.592 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,cd02c69af456] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131741, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116160, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:01.566+08:00, sub_task_instance_id=131741}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:01.597 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:54:01.597 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:54:01.597 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:54:01.597 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:54:01.598 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:54:01.600 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,cd02c69af456] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131741, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:01.601 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:54:01.602 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:54:01.602 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:54:01.603 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:54:01.604 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:54:01.624 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:54:01.627 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:54:01.627 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:54:01.628 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:54:01.628 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:54:01.640 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 13:54:01.641 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 13:54:01.643 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 13:54:01.650 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:54:01.650 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:54:01.651 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:54:01.658 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116159, status:SUCCESS 2023-05-15 13:54:01.659 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116159, status:SUCCESS 2023-05-15 13:54:01.659 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:54:01.665 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:54:01.676 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116160, status:SUCCESS 2023-05-15 13:54:01.677 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116160, status:SUCCESS 2023-05-15 13:54:01.688 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116161, status:SUCCESS 2023-05-15 13:54:01.689 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116161, status:SUCCESS 2023-05-15 13:54:01.700 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116162, status:SUCCESS 2023-05-15 13:54:01.702 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116162, status:SUCCESS 2023-05-15 13:54:01.714 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116163, status:SUCCESS 2023-05-15 13:54:01.716 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116163, status:SUCCESS 2023-05-15 13:54:01.727 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116164, status:SUCCESS 2023-05-15 13:54:01.728 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116164, status:SUCCESS 2023-05-15 13:54:01.754 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 13:54:01.754 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 13:54:01.754 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 13:54:01.754 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 13:54:02.066 INFO 11 --- [pool-ocp-schedules-6,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 13:54:02.426 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:02.427 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:03.176 INFO 11 --- [pool-ocp-schedules-1,,5fc6c4ae220e] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:54:03.176 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:54:03.176 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:54:03.176 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:54:03.177 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:54:03.176 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:54:03.182 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:54:03.183 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:54:03.183 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:54:03.184 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:54:03.184 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:54:03.209 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:54:03.211 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:54:03.212 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:54:03.211718403+08:00 2023-05-15 13:54:03.212 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:54:03.212+08:00 2023-05-15 13:54:03.209 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:54:03.212 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:54:03.213 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:54:03.213148369+08:00 2023-05-15 13:54:03.213 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:54:03.213+08:00 2023-05-15 13:54:03.214 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:54:03.214309836+08:00 2023-05-15 13:54:03.214 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:54:03.214253903+08:00 2023-05-15 13:54:03.214 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:54:03.214+08:00 2023-05-15 13:54:03.214 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:54:03.214+08:00 2023-05-15 13:54:03.216 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:54:03.218 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:54:03.218510825+08:00 2023-05-15 13:54:03.219 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:54:03.219+08:00 2023-05-15 13:54:03.223 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:54:03.213+08:00 and status AVAILABLE 2023-05-15 13:54:03.223 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:54:03.219+08:00 and status AVAILABLE 2023-05-15 13:54:03.226 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:54:03.212+08:00 and status AVAILABLE 2023-05-15 13:54:03.227 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:54:03.214+08:00 and status AVAILABLE 2023-05-15 13:54:03.228 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:54:03.229 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:54:03.232 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:54:03.241 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130043 0 0 , error= 2023-05-15 13:54:03.256 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130043 0 0 , error= 2023-05-15 13:54:03.256 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:54:03.256+08:00 2023-05-15 13:54:03.257 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:54:03.256+08:00 2023-05-15 13:54:03.257 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:54:03.214+08:00 and status AVAILABLE 2023-05-15 13:54:03.257 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:54:03.273 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130043 0 0 , error= 2023-05-15 13:54:03.273 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:54:03.273+08:00 2023-05-15 13:54:03.278 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130043 0 0 , error= 2023-05-15 13:54:03.280 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:54:03.256+08:00 and status AVAILABLE 2023-05-15 13:54:03.280 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:54:03.280+08:00 2023-05-15 13:54:03.281 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:54:03.256+08:00 and status AVAILABLE 2023-05-15 13:54:03.282 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:54:03.284 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:54:03.285 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:54:03.273+08:00 and status AVAILABLE 2023-05-15 13:54:03.285 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:54:03.280+08:00 and status AVAILABLE 2023-05-15 13:54:03.286 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:54:03.287 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:54:03.288 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:54:03.311 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130043 0 0 , error= 2023-05-15 13:54:03.311 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:54:03.311+08:00 2023-05-15 13:54:03.316 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:54:03.311+08:00 and status AVAILABLE 2023-05-15 13:54:03.317 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:54:03.317 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:54:03.318 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:54:03.319 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:54:03.319 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:54:03.328 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:54:03.342 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:54:03.346 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:54:03.347 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:54:03.349 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:54:03.353 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:54:03.391 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:54:03.402 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:54:03.405 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:54:03.407 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:54:03.409 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:54:03.418 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:54:03.420 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:54:03.421 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:54:03.423 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:54:03.434 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:54:03.435 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:54:03.437 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:54:03.438 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:54:03.443 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:54:03.452 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:54:03.481 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:54:03.489 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:54:03.492 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:54:03.496 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:54:03.498 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:54:03.504 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:54:03.506 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:54:03.509 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:54:03.513 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:54:03.522 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:54:03.523 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:54:03.529 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:54:03.531 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:54:03.541 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:54:03.558 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:54:03.559 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:54:03.571 INFO 11 --- [pool-Check host status7,,1d2fa0421311] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:54:03.579 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:54:03.582 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:54:03.582 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:54:03.592 INFO 11 --- [pool-Check host status21,,eb252fe30d5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:54:03.595 INFO 11 --- [pool-Check host status29,,b40611f9f9d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:54:03.596 INFO 11 --- [pool-Check host status4,,41d9830d0b00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:54:03.639 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:54:03.652 INFO 11 --- [pool-Check host status6,,5da275aea8c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:54:03.668 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:54:03.668 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:54:03.668 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:54:03.668 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:54:03.668 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:54:03.674 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:54:03.674 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:54:03.675 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:54:03.675 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:54:03.675 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:54:03.703 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:54:03.704 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:54:03.706 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:54:03.706 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:54:03.708 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status11,,eeb42e180f31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:54:03.723 INFO 11 --- [pool-Check host status20,,f2dc4e8dba7f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:54:03.724 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:54:03.724 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:54:03.724 INFO 11 --- [pool-Check host status27,,b36f21119656] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:54:03.728 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:54:03.728 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:54:03.728 INFO 11 --- [pool-Check host status5,,47af18930bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:54:03.735 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:54:03.735 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:54:03.735 INFO 11 --- [pool-Check host status23,,b15fd8b002db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:54:04.713 INFO 11 --- [pool-ocp-schedules-11,,6592ef615ff2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:54:04.720 INFO 11 --- [pool-ocp-schedules-11,,6592ef615ff2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:54:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:04.830546+08:00, lastProcessTime=2023-05-15T13:53:54.831976+08:00 2023-05-15 13:54:04.859 INFO 11 --- [alarm-detect-15,,a459affd96b0] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130024 ~ 1684130034 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130030=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:04.907 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:04.830546+08:00 2023-05-15 13:54:05.177 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:05.0, nextRun=2023-05-15 13:54:05.0, currentDate=Mon May 15 13:54:05 CST 2023 2023-05-15 13:54:05.177 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:53:05.0, nextRunTime=2023-05-15 13:54:05.0} 2023-05-15 13:54:05.180 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:54:05.180 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:05.180 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:05.427 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:05.429 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:05.649 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131746, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:05.661 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131746, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:54:05.646+08:00, endTime=null} 2023-05-15 13:54:05.691 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,68b8b6c72301] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131746, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116165, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:05.645+08:00, sub_task_instance_id=131746}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:05.718 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,d2c974c0ada3] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:54:05.718 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,12b0490ca534] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:54:05.718 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,870b50df5f0f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:54:05.752 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:05.754 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:05.756 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:05.773 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:05.774 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,d2c974c0ada3] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:54:05.779 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,12b0490ca534] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:54:05.780 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,870b50df5f0f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:54:05.803 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,68b8b6c72301] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131746, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:06.655 INFO 11 --- [pool-ocp-schedules-18,,636eaad5583e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:06.743 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:06.803 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,41c04c2a3ab24dc9,4f27581d180f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=41c04c2a3ab24dc9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:54:06.805 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,41c04c2a3ab24dc9,4f27581d180f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=41c04c2a3ab24dc9, duration=3 ms] 2023-05-15 13:54:06.829 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116165, status:SUCCESS 2023-05-15 13:54:06.830 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116165, status:SUCCESS 2023-05-15 13:54:08.251 INFO 11 --- [pool-overview-container-2,,aa35989f1896] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 13:54:08.251 INFO 11 --- [pool-overview-container-2,,aa35989f1896] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:08.391 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,58f1326ee9dd429d,2db339e59040] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=58f1326ee9dd429d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:08.396 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,58f1326ee9dd429d,2db339e59040] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=58f1326ee9dd429d, duration=5 ms] 2023-05-15 13:54:08.429 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:08.430 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:08.508 INFO 11 --- [pool-scanner--15965401-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:54:09.863 INFO 11 --- [pool-ocp-schedules-7,,7bef6465df8d] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 13:54:11.430 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:11.431 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:11.574 INFO 11 --- [pool-ocp-schedules-1,,f9868afd91d0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:11.574 INFO 11 --- [pool-ocp-schedules-1,,f9868afd91d0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:12.190 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:13.535 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,a7b8c6cff95b4598,8ac57e0549dc] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=a7b8c6cff95b4598, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:13.541 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,a7b8c6cff95b4598,8ac57e0549dc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=a7b8c6cff95b4598, duration=6 ms] 2023-05-15 13:54:14.083 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684129980, nodeCount=500, affectRows=500 2023-05-15 13:54:14.087 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684129980, nodeCount=500, validCount=500 2023-05-15 13:54:14.087 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684129980, nodeCount=335, affectRows=335 2023-05-15 13:54:14.089 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684129980, nodeCount=335, validCount=335 2023-05-15 13:54:14.098 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684129980, nodeCount=1000, affectRows=1000 2023-05-15 13:54:14.104 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684129980, nodeCount=1000, validCount=1000 2023-05-15 13:54:14.110 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684129980, nodeCount=855, affectRows=855 2023-05-15 13:54:14.116 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684129980, nodeCount=855, validCount=855 2023-05-15 13:54:14.431 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:14.433 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:14.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:14.829072+08:00, lastProcessTime=2023-05-15T13:54:04.830546+08:00 2023-05-15 13:54:14.863 INFO 11 --- [alarm-detect-3,,8f32d66dc6d4] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130034 ~ 1684130044 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130040=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:14.865 INFO 11 --- [alarm-detect-3,,8f32d66dc6d4] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 13:54:14.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:14.829072+08:00 2023-05-15 13:54:15.755 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:15.757 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:15.759 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:15.775 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:16.677 INFO 11 --- [pool-ocp-schedules-7,,66b5ad06182d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:16.746 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:17.433 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:17.434 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:18.249 INFO 11 --- [pool-overview-container-7,,fd7503decc6d] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:54:18.250 INFO 11 --- [pool-overview-container-7,,fd7503decc6d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:18.505 INFO 11 --- [pool-scanner--15965501-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:19.427 INFO 11 --- [pool-scanner--15965502-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:19.451 INFO 11 --- [pool-scanner--15965503-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:19.473 INFO 11 --- [pool-scanner--15965504-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:19.499 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.504 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.551 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.555 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.570 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.573 INFO 11 --- [pool-ocp-schedules-14,,2fa49b0e6dc7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:19.869 INFO 11 --- [pool-ocp-schedules-20,,a154139d9327] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:54:20.434 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:20.435 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:20.704 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:21.0, nextRun=2023-05-15 13:54:20.0, currentDate=Mon May 15 13:54:20 CST 2023 2023-05-15 13:54:20.704 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:53:21.0, nextRunTime=2023-05-15 13:54:20.0} 2023-05-15 13:54:20.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:54:20.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:20.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:20.909 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131747, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:20.914 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131747, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:54:20.907+08:00, endTime=null} 2023-05-15 13:54:20.937 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,391e3805403b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131747, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116166, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:20.906+08:00, sub_task_instance_id=131747}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:20.939 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,391e3805403b] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:54:20.940 INFO 11 --- [ForkJoinPool-65957-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:54:20.940 INFO 11 --- [ForkJoinPool-65957-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:54:20.946 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,391e3805403b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131747, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:21.084 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116166, status:SUCCESS 2023-05-15 13:54:21.085 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116166, status:SUCCESS 2023-05-15 13:54:21.749 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:21.751 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:21.754 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:22.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,96b88dae1599468f,f5f6662d6f38] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=96b88dae1599468f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:22.279 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,96b88dae1599468f,f5f6662d6f38] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=96b88dae1599468f, duration=7 ms] 2023-05-15 13:54:23.435 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:23.436 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:24.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:24.832257+08:00, lastProcessTime=2023-05-15T13:54:14.829072+08:00 2023-05-15 13:54:24.878 INFO 11 --- [alarm-detect-13,,dfaeb1be0538] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130044 ~ 1684130054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130050=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:24.880 INFO 11 --- [alarm-detect-5,,a033cadeb1f4] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684129994 ~ 1684130054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388437.988, values={1684130040=388437.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:54:24.881 INFO 11 --- [alarm-detect-5,,a033cadeb1f4] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684129994 ~ 1684130054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993235.358, values={1684130040=993235.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:24.882 INFO 11 --- [alarm-detect-6,,4cb880ef85da] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684129994 ~ 1684130054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388437.938, values={1684130040=388437.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:54:24.883 INFO 11 --- [alarm-detect-6,,4cb880ef85da] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684129994 ~ 1684130054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993235.107, values={1684130040=993235.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:24.911 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:24.832257+08:00 2023-05-15 13:54:25.757 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:25.760 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:25.762 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:25.778 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:25.873 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:26.0, nextRun=2023-05-15 13:54:25.0, currentDate=Mon May 15 13:54:25 CST 2023 2023-05-15 13:54:25.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:53:26.0, nextRunTime=2023-05-15 13:54:25.0} 2023-05-15 13:54:25.885 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:54:25.885 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:25.885 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:25.999 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131748, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:26.000 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131748, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:54:25.996+08:00, endTime=null} 2023-05-15 13:54:26.021 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131748, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116167, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:25.996+08:00, sub_task_instance_id=131748}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:54:26.064 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:54:26.065 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:54:26.066 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:54:26.066 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:54:26.067 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:54:26.072 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe0c83a39a08] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131748, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:26.177 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116167, status:SUCCESS 2023-05-15 13:54:26.178 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116167, status:SUCCESS 2023-05-15 13:54:26.436 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:26.438 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:26.483 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,e49fbf44b5d24a57,852abcab2240] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=e49fbf44b5d24a57, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:26.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,e49fbf44b5d24a57,852abcab2240] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=e49fbf44b5d24a57, duration=7 ms] 2023-05-15 13:54:26.699 INFO 11 --- [pool-ocp-schedules-17,,f33ea9614d4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:26.749 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:27.192 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:27.807 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,165418433f3745d9,a1ef0459bebe] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=165418433f3745d9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:27.825 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,165418433f3745d9,a1ef0459bebe] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=165418433f3745d9, duration=19 ms] 2023-05-15 13:54:28.248 INFO 11 --- [pool-overview-container-1,,f67d5284b149] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:28.503 INFO 11 --- [pool-scanner--15965605-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:29.438 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:29.439 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:30.027 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:53:30.0, nextRun=2023-05-15 13:54:30.0, currentDate=Mon May 15 13:54:30 CST 2023 2023-05-15 13:54:30.027 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:53:30.0, nextRunTime=2023-05-15 13:54:30.0} 2023-05-15 13:54:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:54:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:30.048 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:54:30.0, currentDate=Mon May 15 13:54:30 CST 2023 2023-05-15 13:54:30.048 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:54:30.0} 2023-05-15 13:54:30.051 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:54:30.052 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:54:30.052 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:54:30.073 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131750, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:30.077 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131750, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:54:30.070+08:00, endTime=null} 2023-05-15 13:54:30.092 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131749, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:30.094 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131749, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:54:30.088+08:00, endTime=null} 2023-05-15 13:54:30.101 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131750, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116168, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:30.070+08:00, sub_task_instance_id=131750}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:54:30.123 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131749, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116168, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:30.088+08:00, sub_task_instance_id=131749}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:54:30.167 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.168 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.168 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.168 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.179 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.180 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.180 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:54:30.187 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.190 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.190 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.191 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.195 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.195 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.195 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.196 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.196 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.197 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:54:30.201 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.201 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.205 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.206 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.206 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.206 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.215 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.215 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.216 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.222 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.222 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.223 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.225 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.225 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.226 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.236 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.236 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.236 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.237 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.237 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:54:30.237 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.249 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.249 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.249 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:54:30.252 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.252 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.252 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.263 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.263 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.263 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:54:30.264 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:54:30.274 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.274 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.275 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:54:30.283 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:54:30.291 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.292 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.292 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.292 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.293 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.300 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.300 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.300 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.300 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.301 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.306 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.306 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.307 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.308 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.308 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.308 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.320 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.321 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.321 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.321 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.322 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.323 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.323 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.323 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.327 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.328 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.329 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.329 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.329 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.334 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.334 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.335 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.345 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.346 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.346 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.346 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.347 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.349 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.349 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.349 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.363 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.364 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.364 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.364 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.364 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.365 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.365 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.366 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.373 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.374 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.375 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.375 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.376 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.380 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.381 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.381 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.386 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.386 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.386 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.386 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.386 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.387 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.387 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.387 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.398 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.399 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.399 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.399 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.400 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.401 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.401 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.402 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.404 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.406 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:54:30.414 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:54:30.421 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.421 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.421 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.422 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.422 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.423 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.423 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.423 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:54:30.433 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.433 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.434 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.434 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.434 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.435 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.435 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.435 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:54:30.435 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:54:30.438 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:54:30.440 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.441 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.442 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.442 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.442 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.450 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.450 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.451 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.455 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:30.455 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.455 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.455 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.456 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:30.458 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.458 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.458 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.466 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.468 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.469 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:54:30.469 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:30.470 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.470 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:54:30.470 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:54:30.470 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.470 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.471 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:54:30.480 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.483 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:54:30.484 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.484 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.484 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:54:30.487 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.490 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:54:30.490 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:54:30.490 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:54:30.494 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.496 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c47e554b7593] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131750, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:30.496 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:54:30.496 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:30.496 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:30.497 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:54:30.509 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:54:30.512 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:54:30.512 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:54:30.512 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:54:30.517 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,b01650971aae] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131749, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:31.124 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131751, operation:EXECUTE, state: RUNNING 2023-05-15 13:54:31.125 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131751, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:54:31.121+08:00, endTime=null} 2023-05-15 13:54:31.141 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,bbae162e6824] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131751, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116169, task_operation=execute, latest_execution_start_time=2023-05-15T13:54:31.121+08:00, sub_task_instance_id=131751}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:54:31.141 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,bbae162e6824] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:54:31.162 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:54:31.173 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.173 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.173 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.173 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.173 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.174 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.175 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:54:31.176 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-1,fb27d19334114ed7,3f3ea93abb41] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-2,fb27d19334114ed7,5e092eb292e3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:54:31.177 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-3,fb27d19334114ed7,264e5abdefd4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-4,fb27d19334114ed7,eb904dfdf2c6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.178 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.179 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.179 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.179 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.180 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.184 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.184 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.184 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.184 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.185 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:54:31.185 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:54:31.189 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:54:31.189 INFO 11 --- [pool-3241158-thread-5,fb27d19334114ed7,b1a81e35bb28] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:54:31.189 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:54:31.189 INFO 11 --- [pool-3241158-thread-6,fb27d19334114ed7,a4c88572ba6d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:54:31.190 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,bbae162e6824] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:54:31.195 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,bbae162e6824] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131751, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:54:31.289 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116168, status:SUCCESS 2023-05-15 13:54:31.290 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116168, status:SUCCESS 2023-05-15 13:54:31.300 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116169, status:SUCCESS 2023-05-15 13:54:31.301 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116169, status:SUCCESS 2023-05-15 13:54:31.580 INFO 11 --- [pool-ocp-schedules-13,,b9581a1b8e40] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:31.580 INFO 11 --- [pool-ocp-schedules-13,,b9581a1b8e40] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:31.928 INFO 11 --- [pool-ocp-schedules-11,,5dcacd4e05b6] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:54:31.930 INFO 11 --- [pool-ocp-schedules-11,,5dcacd4e05b6] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:54:31.952 INFO 11 --- [pool-ocp-schedules-4,,984679b6e401] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:54:31.955 INFO 11 --- [pool-ocp-schedules-4,,984679b6e401] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:54:31.956 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.956 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.956 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.957 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.958 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:31.958 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.958 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.959 INFO 11 --- [ForkJoinPool-65958-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:54:31.962 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.963 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.963 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.963 INFO 11 --- [pool-ocp-schedules-14,,ecd028c47794] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:54:31.964 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.965 INFO 11 --- [pool-ocp-schedules-14,,ecd028c47794] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:54:31.969 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:31.969 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:31.970 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:31.974 INFO 11 --- [ForkJoinPool-65958-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:54:32.001 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.001 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.001 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.002 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.011 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.011 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.011 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.019 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.019 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.019 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.020 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:54:32.023 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.024 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.024 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:54:32.074 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:32.075 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.075 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.075 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.076 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.077 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.077 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.078 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:54:32.079 WARN 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:54:32.088 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.088 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.088 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:54:32.089 INFO 11 --- [ForkJoinPool-65960-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:54:32.089 INFO 11 --- [ForkJoinPool-65960-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:54:32.089 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:54:32.102 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.103 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:54:32.106 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:32.107 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.107 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.108 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.108 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.113 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.113 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.114 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:54:32.116 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.116 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.117 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:54:32.119 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:54:32.123 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.137 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.137 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:32.137 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:54:32.137 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.138 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.138 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.138 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.139 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:54:32.139 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.140 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.147 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:54:32.147 INFO 11 --- [ForkJoinPool-65960-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129536321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:54:32.149 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.149 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:54:32.162 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.163 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:54:32.179 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.179 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:54:32.191 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.191 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:54:32.209 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:54:32.226 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:32.227 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.228 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.228 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.229 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.233 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:32.233 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:32.234 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:32.252 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:54:32.252 INFO 11 --- [ForkJoinPool-65960-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129546903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:54:32.439 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:32.440 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:34.836 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:34.836022+08:00, lastProcessTime=2023-05-15T13:54:24.832257+08:00 2023-05-15 13:54:34.853 INFO 11 --- [alarm-detect-10,,afa05386b65d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130054 ~ 1684130064 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130060=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:34.886 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:34.836022+08:00 2023-05-15 13:54:35.441 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:35.442 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:35.760 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:35.762 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:35.764 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:35.780 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:36.718 INFO 11 --- [pool-ocp-schedules-5,,95b44281d6a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:36.753 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:37.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,c525759d6a864c31,918aeb280e82] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=c525759d6a864c31, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:37.891 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,c525759d6a864c31,918aeb280e82] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=c525759d6a864c31, duration=5 ms] 2023-05-15 13:54:38.249 INFO 11 --- [pool-overview-container-5,,59e0cfcea371] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:38.442 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:38.443 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:38.498 INFO 11 --- [pool-scanner--15965713-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:40.860 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,2b24f1e7594a4f01,6cc158c5c0b6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=2b24f1e7594a4f01, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:54:40.862 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,2b24f1e7594a4f01,6cc158c5c0b6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=2b24f1e7594a4f01, duration=2 ms] 2023-05-15 13:54:41.443 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:41.445 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:42.195 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:44.445 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:44.446 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:44.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:44.827923+08:00, lastProcessTime=2023-05-15T13:54:34.836022+08:00 2023-05-15 13:54:44.861 INFO 11 --- [alarm-detect-5,,90a417017999] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130064 ~ 1684130074 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130070=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:44.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:44.827923+08:00 2023-05-15 13:54:45.441 INFO 11 --- [pool-ocp-schedules-3,,63b46a57b16b] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:54:45.763 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:45.765 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:45.767 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:45.783 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:46.570 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568} 2023-05-15 13:54:46.600 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.601 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.602 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.602 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.604 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.604 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.611 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.611 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.616 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.616 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.618 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:50:46.5, end=2023-05-15T13:52:46.568},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:54:46.618 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:54:46.618 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 13:54:46.740 INFO 11 --- [pool-ocp-schedules-6,,24f7ebf39e8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:46.756 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:47.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,71006ca7215042fb,3e9e9e927464] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=71006ca7215042fb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:47.439 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,71006ca7215042fb,3e9e9e927464] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=71006ca7215042fb, duration=6 ms] 2023-05-15 13:54:47.446 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:47.450 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:48.249 INFO 11 --- [pool-overview-container-2,,3815d42aa620] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:48.494 INFO 11 --- [pool-scanner--15965813-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:50.450 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:50.451 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:51.586 INFO 11 --- [pool-ocp-schedules-7,,021dbd51b9bf] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:51.586 INFO 11 --- [pool-ocp-schedules-7,,021dbd51b9bf] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:54:51.751 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:51.759 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:51.764 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:52.674 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:52.724 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:52.731 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 4 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:52.750 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:52.760 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:52.821 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:52.821 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:52.821 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:52.822 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:52.879 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:52.879 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:52.880 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 13:54:53.164 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:54:53.169 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:53.170 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:53.170 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:53.171 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:54:53.175 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:54:53.175 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:54:53.176 INFO 11 --- [pool-sql_perf_diagnoser1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:54:53.240 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.243 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.245 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.247 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.249 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.249 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.261 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.268 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 8 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.288 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 29 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.331 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 113 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.356 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.358 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.361 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.363 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.364 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.365 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.366 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.368 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.371 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.373 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.374 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.376 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.378 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.380 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.381 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:48:52.453, end=2023-05-15T13:53:52.672} 2023-05-15 13:54:53.381 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:54:53.381 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 13:54:53.452 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:53.452 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,216b534b7a44] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40108} 2023-05-15 13:54:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,216b534b7a44] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:54:54.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:54:54.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:54:54.830209+08:00, lastProcessTime=2023-05-15T13:54:44.827923+08:00 2023-05-15 13:54:54.853 INFO 11 --- [alarm-detect-13,,7f90ba735e0e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130074 ~ 1684130084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130080=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:54:54.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:54:54.830209+08:00 2023-05-15 13:54:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,540fc81a8eb9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:54:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,648f5984ee95] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:54:55.765 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:55.768 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:55.770 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:55.772 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,fc015f6d20204ab3,06c8b2f1e436] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=fc015f6d20204ab3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:54:55.777 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,fc015f6d20204ab3,06c8b2f1e436] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=fc015f6d20204ab3, duration=5 ms] 2023-05-15 13:54:55.785 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:56.453 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:56.454 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:54:56.759 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:56.761 INFO 11 --- [pool-ocp-schedules-6,,c382824c51ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:54:57.200 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:54:58.249 INFO 11 --- [pool-overview-container-7,,8aff94069135] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:54:58.501 INFO 11 --- [pool-scanner--15965907-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:54:59.454 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:54:59.455 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:00.000 INFO 11 --- [pool-ocp-schedules-9,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:55:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:55:00.000 INFO 11 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:55:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130100 2023-05-15 13:55:00.034 INFO 11 --- [pool-ocp-schedules-9,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:55:00.984 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:00 CST 2023 2023-05-15 13:55:00.984 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-05-15 13:55:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:00.994 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:00 CST 2023 2023-05-15 13:55:00.994 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.032 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 13:55:01.033 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.033 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.039 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.039 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:55:01.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.050 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.050 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.063 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:55:01.063 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.063 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.068 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.068 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.077 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:55:01.077 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.077 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.082 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:30.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.082 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:54:30.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.087 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:55:01.087 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.087 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.094 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.094 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.098 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-05-15 13:55:01.098 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.098 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.105 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.105 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 13:55:01.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.148 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:55:00.0, currentDate=Mon May 15 13:55:01 CST 2023 2023-05-15 13:55:01.148 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:55:00.0} 2023-05-15 13:55:01.151 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:55:01.151 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:01.151 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,8bece11177e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130160, interval=60 2023-05-15 13:55:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,48c9274a6d26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130160, interval=60 2023-05-15 13:55:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,48c9274a6d26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:55:01.525 INFO 11 --- [pool-prometheus-minute-schedule-2,,48c9274a6d26] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130101, count=6, elapsed=1682445971424 2023-05-15 13:55:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,8bece11177e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:55:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,8bece11177e3] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130101, count=8, elapsed=1682445971424 2023-05-15 13:55:01.555 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131760, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.557 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131760, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:55:01.552+08:00, endTime=null} 2023-05-15 13:55:01.567 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131754, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.569 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131754, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:55:01.565+08:00, endTime=null} 2023-05-15 13:55:01.575 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5cec6db2ef64] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131760, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116178, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.552+08:00, sub_task_instance_id=131760}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.579 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131755, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.581 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131755, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:55:01.577+08:00, endTime=null} 2023-05-15 13:55:01.586 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,730a0a67e227] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131754, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116172, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.565+08:00, sub_task_instance_id=131754}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.589 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5cec6db2ef64] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131760, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.592 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131752, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.594 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131752, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-05-15T13:55:01.589+08:00, endTime=null} 2023-05-15 13:55:01.595 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:55:01.596 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:55:01.596 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:55:01.596 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:55:01.596 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:55:01.600 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:55:01.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4949d21bc228] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131755, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116173, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.577+08:00, sub_task_instance_id=131755}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.601 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:55:01.601 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:55:01.601 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:55:01.602 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:55:01.602 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,730a0a67e227] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131754, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.605 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131753, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.607 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131753, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-05-15T13:55:01.603+08:00, endTime=null} 2023-05-15 13:55:01.614 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,e0d6e055d7b6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131752, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116170, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.589+08:00, sub_task_instance_id=131752}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.616 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,e0d6e055d7b6] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-05-15 13:55:01.621 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4949d21bc228] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:55:01.621 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131758, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.622 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4949d21bc228] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:55:01.623 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,e0d6e055d7b6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131752, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.623 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131758, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-05-15T13:55:01.618+08:00, endTime=null} 2023-05-15 13:55:01.627 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4949d21bc228] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131755, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.627 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:55:01.627 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:55:01.635 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:55:01.635 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:55:01.635 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,485dcde0c98a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131753, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116171, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.603+08:00, sub_task_instance_id=131753}, listMap={cluster_ids=[[1,1,1,2]], tenant_ids=[[2,3,4,6]]}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.635 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,485dcde0c98a] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [2, 3, 4, 6] 2023-05-15 13:55:01.635 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:55:01.638 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131759, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.642 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.642 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.642 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.642 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131759, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-05-15T13:55:01.635+08:00, endTime=null} 2023-05-15 13:55:01.642 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.651 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.652 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.652 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 13:55:01.653 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.654 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.654 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.654 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.655 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,62b938bce2f9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131758, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116176, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.618+08:00, sub_task_instance_id=131758}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.655 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,62b938bce2f9] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-05-15 13:55:01.655 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131756, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.657 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131756, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:55:01.653+08:00, endTime=null} 2023-05-15 13:55:01.659 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.659 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.660 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 13:55:01.662 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131759, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116177, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.635+08:00, sub_task_instance_id=131759}, listMap={cluster_ids=[[1,2,1,1]], tenant_ids=[[3,6,2,4]]}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[3, 6, 2, 4] 2023-05-15 13:55:01.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1684130101663000 2023-05-15 13:55:01.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1684129801170000 2023-05-15 13:55:01.666 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131757, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:01.671 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,196c627d7715444f,325e2618107f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=196c627d7715444f, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=ocpcluster,1,false,2,] 2023-05-15 13:55:01.672 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,196c627d7715444f,325e2618107f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=196c627d7715444f, duration=1 ms] 2023-05-15 13:55:01.673 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131757, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:55:01.664+08:00, endTime=null} 2023-05-15 13:55:01.673 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.673 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.674 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 13:55:01.674 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 13:55:01.674 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-05-15 13:55:01.675 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 13:55:01.675 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131756, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116174, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.653+08:00, sub_task_instance_id=131756}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.679 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1684130101676000 2023-05-15 13:55:01.679 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1684129801182000 2023-05-15 13:55:01.687 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.687 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.687 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.687 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.689 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f06c316f036] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131757, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116175, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:01.664+08:00, sub_task_instance_id=131757}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:01.689 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f06c316f036] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:55:01.692 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.693 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.693 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,485dcde0c98a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131753, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.693 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.696 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.699 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.708 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.709 INFO 11 --- [ForkJoinPool-65963-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.10, 2982] 2023-05-15 13:55:01.709 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,258d53b475164582,4bb8a6a6cfdd] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=258d53b475164582, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-05-15 13:55:01.710 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:55:01.710 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:55:01.710 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,258d53b475164582,4bb8a6a6cfdd] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=258d53b475164582, duration=1 ms] 2023-05-15 13:55:01.711 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 13:55:01.711 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 13:55:01.712 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.712 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.713 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 13:55:01.713 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2023-05-15 13:55:01.714 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.714 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.714 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130101000000] 2023-05-15 13:55:01.717 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67bed9b9c7a2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131756, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.717 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.717 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.718 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1684130101715000 2023-05-15 13:55:01.718 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1684129801253000 2023-05-15 13:55:01.718 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130101000000] 2023-05-15 13:55:01.723 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,8cc8fa125b504b43,ff9b36b5c6ad] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8cc8fa125b504b43, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:01.728 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,8cc8fa125b504b43,ff9b36b5c6ad] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8cc8fa125b504b43, duration=5 ms] 2023-05-15 13:55:01.731 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-05-15 13:55:01.732 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:55:01.733 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:55:01.733 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:55:01.733 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:55:01.733 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:55:01.733 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:55:01.735 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1684130101732000 2023-05-15 13:55:01.735 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1684129801240000 2023-05-15 13:55:01.741 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.741 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.743 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.11, 2982] 2023-05-15 13:55:01.745 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.745 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.745 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.745 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.745 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.745 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.745 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.746 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.746 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.746 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.13, 2982] 2023-05-15 13:55:01.746 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.746 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.746 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.747 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2023-05-15 13:55:01.747 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.748 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.748 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.748 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.748 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.749 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.749 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.749 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.749 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.749 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:01.749 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.750 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:55:01.750 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:55:01.750 INFO 11 --- [ForkJoinPool-65963-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.12, 2982] 2023-05-15 13:55:01.751 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:55:01.751 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:55:01.751 INFO 11 --- [pool-3241326-thread-3,fb27d19334114ed7,680de573fe7a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:55:01.751 INFO 11 --- [pool-3241326-thread-1,fb27d19334114ed7,038d2a29893f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:55:01.751 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.752 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.752 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.752 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:55:01.752 INFO 11 --- [pool-3241326-thread-4,fb27d19334114ed7,94b61a1d05ae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-2,fb27d19334114ed7,2f8930c3febb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:55:01.753 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.754 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,11161985ae51] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131759, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.754 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:01.773 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.773 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.773 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:01.773 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:01.774 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:55:01.774 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:55:01.780 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:55:01.780 INFO 11 --- [pool-3241326-thread-5,fb27d19334114ed7,e7d994cdf936] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:55:01.782 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:55:01.783 INFO 11 --- [pool-3241326-thread-6,fb27d19334114ed7,a337e4a92769] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:55:01.784 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f06c316f036] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:55:01.790 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f06c316f036] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131757, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:01.847 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116170, status:SUCCESS 2023-05-15 13:55:01.848 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116170, status:SUCCESS 2023-05-15 13:55:01.854 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116171, status:SUCCESS 2023-05-15 13:55:01.855 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116171, status:SUCCESS 2023-05-15 13:55:01.863 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116172, status:SUCCESS 2023-05-15 13:55:01.864 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116172, status:SUCCESS 2023-05-15 13:55:01.871 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116173, status:SUCCESS 2023-05-15 13:55:01.872 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116173, status:SUCCESS 2023-05-15 13:55:01.886 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116174, status:SUCCESS 2023-05-15 13:55:01.887 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116174, status:SUCCESS 2023-05-15 13:55:01.896 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116175, status:SUCCESS 2023-05-15 13:55:01.897 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116175, status:SUCCESS 2023-05-15 13:55:01.909 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116177, status:SUCCESS 2023-05-15 13:55:01.910 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116177, status:SUCCESS 2023-05-15 13:55:01.919 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116178, status:SUCCESS 2023-05-15 13:55:01.920 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116178, status:SUCCESS 2023-05-15 13:55:01.951 INFO 11 --- [pool-3241325-thread-1,fb27d19334114ed7,48035ef67f61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 13:55:02.016 WARN 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obcluster, ob tenant id:1001 2023-05-15 13:55:02.018 INFO 11 --- [pool-3241325-thread-2,fb27d19334114ed7,107bbb39f7ea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 13:55:02.020 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,62b938bce2f9] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-05-15 13:55:02.033 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,62b938bce2f9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131758, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:02.455 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:02.457 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:02.959 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116176, status:SUCCESS 2023-05-15 13:55:02.960 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116176, status:SUCCESS 2023-05-15 13:55:03.749 INFO 11 --- [pool-ocp-schedules-11,,4215e8ecc61d] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:55:03.749 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:55:03.749 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:55:03.749 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:55:03.749 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:55:03.749 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:55:03.756 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:55:03.756 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:55:03.756 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:55:03.756 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:55:03.757 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:55:03.788 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:55:03.789 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:55:03.789 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:55:03.790 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:55:03.790 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:55:03.790017454+08:00 2023-05-15 13:55:03.790 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:55:03.790+08:00 2023-05-15 13:55:03.791 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:55:03.790773358+08:00 2023-05-15 13:55:03.791 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:55:03.791+08:00 2023-05-15 13:55:03.791 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:55:03.790804511+08:00 2023-05-15 13:55:03.791 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:55:03.791+08:00 2023-05-15 13:55:03.791 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:55:03.792 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:55:03.792225854+08:00 2023-05-15 13:55:03.792 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:55:03.792+08:00 2023-05-15 13:55:03.793 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:55:03.793510974+08:00 2023-05-15 13:55:03.794 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:55:03.793+08:00 2023-05-15 13:55:03.794 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:55:03.791+08:00 and status AVAILABLE 2023-05-15 13:55:03.794 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:55:03.790+08:00 and status AVAILABLE 2023-05-15 13:55:03.797 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:55:03.792+08:00 and status AVAILABLE 2023-05-15 13:55:03.797 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:55:03.791+08:00 and status AVAILABLE 2023-05-15 13:55:03.798 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:55:03.793+08:00 and status AVAILABLE 2023-05-15 13:55:03.800 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:55:03.802 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:55:03.824 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130103 0 0 , error= 2023-05-15 13:55:03.827 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:55:03.824+08:00 2023-05-15 13:55:03.830 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130103 0 0 , error= 2023-05-15 13:55:03.830 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:55:03.830 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:55:03.830+08:00 2023-05-15 13:55:03.831 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:55:03.831 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:55:03.840 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130103 0 0 , error= 2023-05-15 13:55:03.843 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:55:03.824+08:00 and status AVAILABLE 2023-05-15 13:55:03.847 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130103 0 0 , error= 2023-05-15 13:55:03.844 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:55:03.830+08:00 and status AVAILABLE 2023-05-15 13:55:03.865 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:55:03.847+08:00 2023-05-15 13:55:03.868 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130103 0 0 , error= 2023-05-15 13:55:03.868 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:55:03.865+08:00 2023-05-15 13:55:03.869 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:55:03.869+08:00 2023-05-15 13:55:03.870 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:55:03.871 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:55:03.873 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:55:03.865+08:00 and status AVAILABLE 2023-05-15 13:55:03.873 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:55:03.847+08:00 and status AVAILABLE 2023-05-15 13:55:03.874 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:55:03.869+08:00 and status AVAILABLE 2023-05-15 13:55:03.876 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:55:03.876 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:55:03.877 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:55:03.883 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:55:03.884 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:55:03.886 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:55:03.888 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:55:03.889 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:55:03.911 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:55:03.911 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:55:03.911 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:55:03.912 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:55:03.913 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:55:03.960 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:55:03.967 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:55:03.971 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:55:03.974 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:55:03.979 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:55:03.983 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:55:03.986 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:55:03.988 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:55:03.992 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:55:03.997 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:55:04.003 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:55:04.004 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:55:04.005 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:55:04.006 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:55:04.024 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:55:04.040 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:55:04.053 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:55:04.054 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:55:04.062 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:55:04.065 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:55:04.069 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:55:04.078 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:55:04.078 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:55:04.079 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:55:04.089 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:55:04.098 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:55:04.099 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:55:04.117 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:55:04.125 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:55:04.132 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:55:04.136 INFO 11 --- [pool-Check host status1,,75aef77bdd3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:55:04.145 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:55:04.152 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:55:04.153 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:55:04.159 INFO 11 --- [pool-Check host status13,,b6a0e3804af0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:55:04.164 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:55:04.166 INFO 11 --- [pool-Check host status14,,b29799f31e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:55:04.177 INFO 11 --- [pool-Check host status8,,53fe60727707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:55:04.249 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:55:04.263 INFO 11 --- [pool-Check host status30,,bbf4c14faa4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:55:04.284 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:55:04.284 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:55:04.284 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:55:04.285 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:55:04.285 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:55:04.291 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:55:04.291 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:55:04.291 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:55:04.291 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:55:04.292 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:55:04.322 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:55:04.322 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:55:04.322 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:55:04.324 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:55:04.338 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:55:04.341 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:55:04.342 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:55:04.342 INFO 11 --- [pool-Check host status17,,08e382eddd5a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:55:04.342 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:55:04.342 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:55:04.342 INFO 11 --- [pool-Check host status3,,54baaa484b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:55:04.343 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:55:04.344 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:55:04.344 INFO 11 --- [pool-Check host status28,,664d8748191d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:55:04.349 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:55:04.349 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:55:04.349 INFO 11 --- [pool-Check host status2,,e4a522da5ba6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:55:04.360 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:55:04.360 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:55:04.360 INFO 11 --- [pool-Check host status19,,c6533530e891] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:55:04.713 INFO 11 --- [pool-ocp-schedules-20,,412bbd554014] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:55:04.720 INFO 11 --- [pool-ocp-schedules-20,,412bbd554014] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:55:04.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:04.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:04.829113+08:00, lastProcessTime=2023-05-15T13:54:54.830209+08:00 2023-05-15 13:55:04.856 INFO 11 --- [alarm-detect-3,,7a60d693e365] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130084 ~ 1684130094 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130090=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:04.890 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:04.829113+08:00 2023-05-15 13:55:05.260 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:05.0, nextRun=2023-05-15 13:55:05.0, currentDate=Mon May 15 13:55:05 CST 2023 2023-05-15 13:55:05.261 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:54:05.0, nextRunTime=2023-05-15 13:55:05.0} 2023-05-15 13:55:05.265 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:55:05.265 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:05.265 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:05.457 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:05.458 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:05.740 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131761, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:05.742 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131761, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:55:05.738+08:00, endTime=null} 2023-05-15 13:55:05.763 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,374dbd601b6a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131761, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116179, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:05.738+08:00, sub_task_instance_id=131761}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:05.768 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:05.770 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:05.773 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:05.787 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:05.787 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,281a23d4e98f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:55:05.790 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,12e594ab0586] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:55:05.792 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,9f433493eaa7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:55:05.830 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,12e594ab0586] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:55:05.832 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,9f433493eaa7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:55:05.832 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,281a23d4e98f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:55:05.844 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,374dbd601b6a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131761, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:06.028 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116179, status:SUCCESS 2023-05-15 13:55:06.029 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116179, status:SUCCESS 2023-05-15 13:55:06.762 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:06.779 INFO 11 --- [pool-ocp-schedules-10,,09d6d283ac30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:08.250 INFO 11 --- [pool-overview-container-1,,5f7c515bf3e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:08.458 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:08.459 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:08.500 INFO 11 --- [pool-scanner--15966000-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:55:09.941 INFO 11 --- [pool-ocp-schedules-1,,24424c50c5c3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40108} 2023-05-15 13:55:09.942 INFO 11 --- [pool-ocp-schedules-1,,24424c50c5c3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:55:11.459 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:11.460 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:11.590 INFO 11 --- [pool-ocp-schedules-6,,63176525ca34] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:11.590 INFO 11 --- [pool-ocp-schedules-6,,63176525ca34] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:12.202 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:13.859 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,e6aed32d0af047a8,4283f5978116] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=e6aed32d0af047a8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:13.864 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,e6aed32d0af047a8,4283f5978116] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=e6aed32d0af047a8, duration=4 ms] 2023-05-15 13:55:14.110 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130040, nodeCount=1000, affectRows=1000 2023-05-15 13:55:14.119 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130040, nodeCount=1000, validCount=1000 2023-05-15 13:55:14.121 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130040, nodeCount=845, affectRows=845 2023-05-15 13:55:14.126 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130040, nodeCount=845, affectRows=845 2023-05-15 13:55:14.127 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130040, nodeCount=845, validCount=845 2023-05-15 13:55:14.132 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130040, nodeCount=845, validCount=845 2023-05-15 13:55:14.461 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:14.462 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:14.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:14.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:14.826924+08:00, lastProcessTime=2023-05-15T13:55:04.829113+08:00 2023-05-15 13:55:14.864 INFO 11 --- [alarm-detect-8,,10a328c0b746] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130094 ~ 1684130104 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130100=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:14.892 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,136a267cf9344091,80eb2ed08161] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=136a267cf9344091, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:55:14.893 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,136a267cf9344091,80eb2ed08161] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=136a267cf9344091, duration=2 ms] 2023-05-15 13:55:14.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:14.826924+08:00 2023-05-15 13:55:15.770 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:15.779 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:15.794 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:15.797 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:16.765 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:16.798 INFO 11 --- [pool-ocp-schedules-12,,2862b864faea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:17.462 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:17.463 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 13:55:17.633 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:17.634 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:17.634 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:17.635 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:17.635 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:55:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:17.672 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:17.672 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:17.673 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:55:17.675 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 13:55:17.676 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 13:55:17.676 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 13:55:17.676 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 13:55:17.680 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=0) - not enough cached interval (source=2023-05-15T13:52:48, target=2023-05-15T13:50:47, granule=120000ms, delay=121000ms) 2023-05-15 13:55:17.680 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-05-15T13:52:48, target=2023-05-15T13:50:47, granule=120000ms, delay=121000ms) 2023-05-15 13:55:17.681 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=0) - not enough cached interval (source=2023-05-15T13:53:02, target=2023-05-15T13:50:47, granule=120000ms, delay=135000ms) 2023-05-15 13:55:17.681 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:50:47, target=2023-05-15T13:39:17, granule=600000ms, delay=690000ms) 2023-05-15 13:55:17.681 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:50:47, target=2023-05-15T13:39:17, granule=600000ms, delay=690000ms) 2023-05-15 13:55:17.681 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:50:47, target=2023-05-15T13:41:17, granule=600000ms, delay=570000ms) 2023-05-15 13:55:17.681 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=94ms, period=120000ms 2023-05-15 13:55:18.258 INFO 11 --- [pool-overview-container-5,,f66f8b8644bf] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:18.500 INFO 11 --- [pool-scanner--15966100-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:18.996 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,c1a996fb50a04cf8,2c78175a5be5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=c1a996fb50a04cf8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:19.003 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,c1a996fb50a04cf8,2c78175a5be5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=c1a996fb50a04cf8, duration=7 ms] 2023-05-15 13:55:19.617 INFO 11 --- [pool-scanner--15966101-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:19.640 INFO 11 --- [pool-scanner--15966102-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:19.657 INFO 11 --- [pool-scanner--15966103-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:19.674 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:19.678 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:19.722 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:19.726 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:19.746 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:19.750 INFO 11 --- [pool-ocp-schedules-20,,2f79b3c528d0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:20.463 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:20.464 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:20.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:21.0, nextRun=2023-05-15 13:55:20.0, currentDate=Mon May 15 13:55:20 CST 2023 2023-05-15 13:55:20.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:54:21.0, nextRunTime=2023-05-15 13:55:20.0} 2023-05-15 13:55:20.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:55:20.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:20.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:20.957 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131762, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:20.959 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131762, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:55:20.953+08:00, endTime=null} 2023-05-15 13:55:20.982 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,443392a83342] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131762, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116180, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:20.953+08:00, sub_task_instance_id=131762}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:20.984 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,443392a83342] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:55:20.984 INFO 11 --- [ForkJoinPool-65966-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:55:20.985 INFO 11 --- [ForkJoinPool-65966-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:55:20.990 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,443392a83342] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131762, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:21.319 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116180, status:SUCCESS 2023-05-15 13:55:21.321 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116180, status:SUCCESS 2023-05-15 13:55:21.330 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,28826c52151845e6,adf30c487351] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=28826c52151845e6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:21.334 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,28826c52151845e6,adf30c487351] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=28826c52151845e6, duration=5 ms] 2023-05-15 13:55:21.467 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-05-15 13:55:21.493 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:55:21.520 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:55:21.545 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:55:21.564 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:55:21.579 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:55:21.602 INFO 11 --- [pool-ocp-schedules-3,,e468c2258816] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=5 2023-05-15 13:55:21.759 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:21.764 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:21.766 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:23.464 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:23.466 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:24.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:24.827928+08:00, lastProcessTime=2023-05-15T13:55:14.826924+08:00 2023-05-15 13:55:24.854 INFO 11 --- [alarm-detect-9,,b59a111d7397] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130104 ~ 1684130114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130110=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:24.856 INFO 11 --- [alarm-detect-3,,38fe06dc028f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130054 ~ 1684130114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388497.988, values={1684130100=388497.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:55:24.858 INFO 11 --- [alarm-detect-3,,38fe06dc028f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130054 ~ 1684130114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993295.358, values={1684130100=993295.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:24.859 INFO 11 --- [alarm-detect-15,,9e452f4f486b] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130054 ~ 1684130114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388497.938, values={1684130100=388497.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:55:24.861 INFO 11 --- [alarm-detect-15,,9e452f4f486b] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130054 ~ 1684130114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993295.107, values={1684130100=993295.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:24.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:24.827928+08:00 2023-05-15 13:55:25.779 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:25.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:25.797 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:25.799 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:25.884 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:26.0, nextRun=2023-05-15 13:55:25.0, currentDate=Mon May 15 13:55:25 CST 2023 2023-05-15 13:55:25.884 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:54:26.0, nextRunTime=2023-05-15 13:55:25.0} 2023-05-15 13:55:25.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:55:25.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:25.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:26.047 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131763, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:26.049 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131763, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:55:26.044+08:00, endTime=null} 2023-05-15 13:55:26.071 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131763, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116181, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:26.043+08:00, sub_task_instance_id=131763}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:55:26.115 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:55:26.116 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:55:26.117 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:55:26.118 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:55:26.119 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:55:26.124 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,573bf8f95dc9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131763, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:26.420 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116181, status:SUCCESS 2023-05-15 13:55:26.422 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116181, status:SUCCESS 2023-05-15 13:55:26.466 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:26.467 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:26.768 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:26.820 INFO 11 --- [pool-ocp-schedules-9,,eca4a34f3147] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:27.205 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:28.250 INFO 11 --- [pool-overview-container-3,,02967eddf55b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:28.504 INFO 11 --- [pool-scanner--15966204-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:29.467 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:29.476 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:30.018 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:30.0, nextRun=2023-05-15 13:55:30.0, currentDate=Mon May 15 13:55:30 CST 2023 2023-05-15 13:55:30.018 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:54:30.0, nextRunTime=2023-05-15 13:55:30.0} 2023-05-15 13:55:30.026 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:55:30.026 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:30.027 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:30.027 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:30.035 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.041 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 13:55:30.0, currentDate=Mon May 15 13:55:30 CST 2023 2023-05-15 13:55:30.041 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 13:55:30.0} 2023-05-15 13:55:30.044 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:55:30.044 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:55:30.045 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:55:30.062 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.064 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.066 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.067 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.069 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.070 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.072 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.073 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.084 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.090 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.096 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.106 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.109 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.111 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.113 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.115 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.117 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131766, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:30.120 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131766, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:55:30.115+08:00, endTime=null} 2023-05-15 13:55:30.124 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.129 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.134 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.135 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131764, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:30.137 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131764, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:55:30.132+08:00, endTime=null} 2023-05-15 13:55:30.139 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.140 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5219314d77e1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131766, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116183, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:30.115+08:00, sub_task_instance_id=131766}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:55:30.140 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5219314d77e1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:55:30.141 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.143 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.145 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.147 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:49:29.911, end=2023-05-15T13:54:30.031} 2023-05-15 13:55:30.147 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 13:55:30.148 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131765, operation:EXECUTE, state: RUNNING 2023-05-15 13:55:30.150 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131765, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:55:30.146+08:00, endTime=null} 2023-05-15 13:55:30.156 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131764, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116182, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:30.131+08:00, sub_task_instance_id=131764}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:55:30.160 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:55:30.172 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131765, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116182, task_operation=execute, latest_execution_start_time=2023-05-15T13:55:30.146+08:00, sub_task_instance_id=131765}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:55:30.172 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.173 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.174 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.175 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:55:30.176 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-3,fb27d19334114ed7,d377a8ff42cd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-1,fb27d19334114ed7,8671062bda48] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-4,fb27d19334114ed7,2444a5366b46] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:55:30.178 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:55:30.179 INFO 11 --- [pool-3241486-thread-2,fb27d19334114ed7,3188969064ad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:55:30.179 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.179 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.179 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.180 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.181 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.181 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.181 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.182 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.185 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.185 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.186 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:55:30.186 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.186 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.187 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:55:30.191 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:55:30.191 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:55:30.191 INFO 11 --- [pool-3241486-thread-5,fb27d19334114ed7,79d18999fa4c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:55:30.191 INFO 11 --- [pool-3241486-thread-6,fb27d19334114ed7,acfb3e6c02d8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:55:30.192 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5219314d77e1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:55:30.192 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.193 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.193 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.193 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:55:30.197 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,5219314d77e1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131766, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:30.200 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.200 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.201 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.213 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.213 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.213 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.224 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.224 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.224 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.225 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.226 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.230 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.230 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:55:30.235 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.236 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.236 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.238 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.238 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.238 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.249 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.249 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.250 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.251 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.251 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.251 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:55:30.261 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.261 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.262 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.265 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.265 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.266 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:55:30.274 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.275 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.275 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.279 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.280 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.280 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:55:30.288 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:55:30.290 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.291 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.291 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:55:30.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:55:30.319 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.319 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.319 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.319 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.320 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.321 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.321 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.321 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.321 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.322 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.323 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.323 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.323 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.328 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.328 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.329 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.331 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.344 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.345 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.345 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.346 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.346 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.349 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.350 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.351 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.351 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.351 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.351 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.360 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.361 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.361 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.361 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.361 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.362 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.362 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.362 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.375 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.376 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.377 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.377 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.377 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.377 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.377 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.378 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.378 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.378 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.381 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.381 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.381 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.397 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.398 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.399 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.399 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.400 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.404 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.404 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.405 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.406 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:55:30.409 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:55:30.410 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.412 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.413 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.413 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.413 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:55:30.427 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.427 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.428 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.428 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.428 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.429 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.429 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.430 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:55:30.430 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:55:30.433 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:55:30.445 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.446 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.447 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.447 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.448 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.454 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.455 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.456 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:30.458 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.459 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.459 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.467 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.470 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:55:30.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:30.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:55:30.474 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.477 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:55:30.477 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:55:30.489 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.490 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.492 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:55:30.492 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:55:30.492 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:55:30.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:55:30.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.494 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:55:30.498 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a1d613add939] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131765, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:30.504 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.506 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:55:30.506 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:30.506 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:30.507 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:55:30.515 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:55:30.515 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116183, status:SUCCESS 2023-05-15 13:55:30.516 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116183, status:SUCCESS 2023-05-15 13:55:30.517 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:55:30.517 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:55:30.517 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:55:30.522 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5aaa450b17ff] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131764, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:55:31.545 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116182, status:SUCCESS 2023-05-15 13:55:31.547 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116182, status:SUCCESS 2023-05-15 13:55:31.597 INFO 11 --- [pool-ocp-schedules-17,,e7bea66802ed] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:31.597 INFO 11 --- [pool-ocp-schedules-17,,e7bea66802ed] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:31.937 INFO 11 --- [pool-ocp-schedules-16,,ac0e512418f0] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:55:31.939 INFO 11 --- [pool-ocp-schedules-16,,ac0e512418f0] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:55:31.957 INFO 11 --- [pool-ocp-schedules-3,,72cd3fe419aa] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:55:31.959 INFO 11 --- [pool-ocp-schedules-3,,72cd3fe419aa] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:55:31.963 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:31.963 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:31.963 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.963 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.964 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.964 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.964 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:31.964 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:31.965 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.965 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.965 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:31.965 INFO 11 --- [ForkJoinPool-65967-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:55:31.968 INFO 11 --- [pool-ocp-schedules-14,,9306d23e941f] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:55:31.968 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:31.968 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.969 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:31.969 INFO 11 --- [pool-ocp-schedules-14,,9306d23e941f] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:55:31.972 INFO 11 --- [ForkJoinPool-65967-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:55:31.989 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.989 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.989 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.989 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:31.990 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:31.990 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:31.990 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:55:32.015 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.016 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.016 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.017 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.021 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.021 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.021 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:55:32.041 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:32.042 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.042 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.042 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.042 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.044 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.044 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.044 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:55:32.045 WARN 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:55:32.046 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.046 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.046 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:55:32.047 INFO 11 --- [ForkJoinPool-65969-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:55:32.048 INFO 11 --- [ForkJoinPool-65969-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:55:32.048 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:55:32.059 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.059 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:55:32.071 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.083 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:32.084 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.084 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.084 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.084 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.086 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:55:32.086 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.086 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.092 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:32.093 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.094 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.094 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.095 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.097 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:55:32.097 INFO 11 --- [ForkJoinPool-65969-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129536321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:55:32.099 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.099 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.100 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:55:32.104 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.104 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.104 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:55:32.107 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:55:32.126 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.126 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:55:32.145 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.145 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:55:32.163 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.163 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:55:32.182 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.182 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:55:32.198 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.199 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:55:32.216 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:55:32.235 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:55:32.237 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.237 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.237 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.238 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.243 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:55:32.243 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:55:32.243 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:55:32.265 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:55:32.265 INFO 11 --- [ForkJoinPool-65969-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684129546903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:55:32.477 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:32.478 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:34.622 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,484b080c9a8140ac,8f63edf983a7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=484b080c9a8140ac, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:34.626 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,484b080c9a8140ac,8f63edf983a7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=484b080c9a8140ac, duration=5 ms] 2023-05-15 13:55:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:34.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:34.829705+08:00, lastProcessTime=2023-05-15T13:55:24.827928+08:00 2023-05-15 13:55:34.859 INFO 11 --- [alarm-detect-8,,52e00f88b126] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130114 ~ 1684130124 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130120=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:34.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:34.829705+08:00 2023-05-15 13:55:35.478 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:35.479 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:35.621 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,ebd0fcee55894de6,3a9022bf1482] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=ebd0fcee55894de6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:35.626 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,ebd0fcee55894de6,3a9022bf1482] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=ebd0fcee55894de6, duration=6 ms] 2023-05-15 13:55:35.781 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:35.796 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:35.800 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:35.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:36.320 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,ef7ba0cbb2884307,831de709b440] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&version=2&ObClusterId=1, client=10.xxx.xx.10, traceId=ef7ba0cbb2884307, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=ocpcluster,1,2,{"ObRegion":"ocpcluster","ObCluster":"ocpcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684130136316107,"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 13:55:36.324 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,ef7ba0cbb2884307,831de709b440] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=ocpcluster, clusterId=1, affectRows=1 2023-05-15 13:55:36.338 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,ef7ba0cbb2884307,831de709b440] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.10, traceId=ef7ba0cbb2884307, duration=19 ms] 2023-05-15 13:55:36.771 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:36.841 INFO 11 --- [pool-ocp-schedules-15,,26f81c88ecfa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:38.250 INFO 11 --- [pool-overview-container-7,,2063911d7f4c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:38.479 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:38.480 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:38.502 INFO 11 --- [pool-scanner--15966312-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:41.481 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:41.482 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:42.211 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:43.144 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,57e0bd66c18d4ee3,d13650fbd439] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=57e0bd66c18d4ee3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:43.151 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,57e0bd66c18d4ee3,d13650fbd439] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=57e0bd66c18d4ee3, duration=7 ms] 2023-05-15 13:55:44.482 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:44.483 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:44.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:44.832135+08:00, lastProcessTime=2023-05-15T13:55:34.829705+08:00 2023-05-15 13:55:44.881 INFO 11 --- [alarm-detect-6,,ed50b33cb1be] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130124 ~ 1684130134 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130130=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:44.910 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:44.832135+08:00 2023-05-15 13:55:45.449 INFO 11 --- [pool-ocp-schedules-15,,01a3cd107be9] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:55:45.783 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:45.800 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:45.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:45.804 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:46.774 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:46.861 INFO 11 --- [pool-ocp-schedules-11,,b1f4d542a15b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:46.902 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0911d86ce7b240c7,df2bf7b4454f] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&version=2&ObClusterId=1, client=10.xxx.xx.11, traceId=0911d86ce7b240c7, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obcluster,1,2,{"ObRegion":"obcluster","ObCluster":"obcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684130146900385,"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 13:55:46.906 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0911d86ce7b240c7,df2bf7b4454f] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=obcluster, clusterId=1, affectRows=1 2023-05-15 13:55:46.907 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0911d86ce7b240c7,df2bf7b4454f] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.11, traceId=0911d86ce7b240c7, duration=5 ms] 2023-05-15 13:55:47.484 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:47.485 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:48.250 INFO 11 --- [pool-overview-container-8,,6388464e1dc6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:48.503 INFO 11 --- [pool-scanner--15966412-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:48.922 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,5a76a56fb8ad4af7,107d41e579f3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=5a76a56fb8ad4af7, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:55:48.923 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,5a76a56fb8ad4af7,107d41e579f3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=5a76a56fb8ad4af7, duration=2 ms] 2023-05-15 13:55:50.485 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:50.486 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:51.603 INFO 11 --- [pool-ocp-schedules-12,,59ca68b42473] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:51.603 INFO 11 --- [pool-ocp-schedules-12,,59ca68b42473] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:55:51.762 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:51.766 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:51.768 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:53.486 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:53.487 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,c6834f04a075] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40108} 2023-05-15 13:55:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,c6834f04a075] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:55:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:55:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:55:54.833447+08:00, lastProcessTime=2023-05-15T13:55:44.832135+08:00 2023-05-15 13:55:54.866 INFO 11 --- [alarm-detect-4,,c2e48c92b156] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130134 ~ 1684130144 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130140=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:55:54.901 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:55:54.833447+08:00 2023-05-15 13:55:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,8a416792a148] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:55:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,81f8d9dbbe19] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:55:55.225 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,0bc5d85736664cfa,848ba8957900] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=0bc5d85736664cfa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:55:55.231 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,0bc5d85736664cfa,848ba8957900] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=0bc5d85736664cfa, duration=6 ms] 2023-05-15 13:55:55.786 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:55.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:55.804 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:55.806 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:56.487 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:56.489 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:55:56.778 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:56.882 INFO 11 --- [pool-ocp-schedules-9,,cb125c6fcae5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:55:57.213 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:55:58.251 INFO 11 --- [pool-overview-container-6,,05a9ac57f2ac] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:55:58.502 INFO 11 --- [pool-scanner--15966506-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:55:59.489 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:55:59.490 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:56:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:56:00.001 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:56:00.001 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130160 2023-05-15 13:56:00.040 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:56:00.916 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:54:01.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.917 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 13:54:01.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:00.926 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 13:56:00.926 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.926 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.926 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:00.937 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.937 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:00.940 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:56:00.940 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.940 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:00.945 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.945 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:00.961 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:56:00.961 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.961 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:00.966 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.966 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:56:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.978 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:00.985 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:30.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.985 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:55:30.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:56:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:00.988 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:00.996 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 13:56:00.0, currentDate=Mon May 15 13:56:00 CST 2023 2023-05-15 13:56:00.996 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 13:56:00.0} 2023-05-15 13:56:01.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:56:01.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:01.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:01.247 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,ca22145d99614ae9,982a74727e23] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ca22145d99614ae9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:01.253 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,ca22145d99614ae9,982a74727e23] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ca22145d99614ae9, duration=6 ms] 2023-05-15 13:56:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,1e2b59e361d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130220, interval=60 2023-05-15 13:56:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,1088896a2f24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130220, interval=60 2023-05-15 13:56:01.521 INFO 11 --- [pool-prometheus-minute-schedule-2,,1088896a2f24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:56:01.521 INFO 11 --- [pool-prometheus-minute-schedule-2,,1088896a2f24] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130161, count=6, elapsed=1682446031360 2023-05-15 13:56:01.554 INFO 11 --- [pool-prometheus-minute-schedule-1,,1e2b59e361d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:56:01.554 INFO 11 --- [pool-prometheus-minute-schedule-1,,1e2b59e361d7] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130161, count=8, elapsed=1682446031393 2023-05-15 13:56:01.626 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131767, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.628 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131767, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T13:56:01.624+08:00, endTime=null} 2023-05-15 13:56:01.645 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131770, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.646 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131770, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:56:01.643+08:00, endTime=null} 2023-05-15 13:56:01.651 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bf41b754ef68] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131767, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116184, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.624+08:00, sub_task_instance_id=131767}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.654 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131771, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.656 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131771, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:56:01.652+08:00, endTime=null} 2023-05-15 13:56:01.661 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1dd9968a055e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131770, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116186, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.643+08:00, sub_task_instance_id=131770}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.667 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131768, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.670 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131768, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T13:56:01.664+08:00, endTime=null} 2023-05-15 13:56:01.673 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bf41b754ef68] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131767, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.675 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,6ae6282f9d55] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131771, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116187, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.652+08:00, sub_task_instance_id=131771}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.681 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131769, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.682 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131769, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:56:01.678+08:00, endTime=null} 2023-05-15 13:56:01.684 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1dd9968a055e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:56:01.684 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1dd9968a055e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:56:01.688 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,df93f94ac2ee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131768, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116184, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.663+08:00, sub_task_instance_id=131768}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.688 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1dd9968a055e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131770, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.691 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131772, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.692 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131772, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:56:01.689+08:00, endTime=null} 2023-05-15 13:56:01.693 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,6ae6282f9d55] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:56:01.693 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,6ae6282f9d55] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:56:01.695 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,6ae6282f9d55] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 13:56:01.697 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,0b7c14fc4453] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131769, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116185, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.678+08:00, sub_task_instance_id=131769}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.697 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,df93f94ac2ee] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131768, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.699 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,6ae6282f9d55] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131771, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.701 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131773, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:01.702 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:56:01.702 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:56:01.702 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:56:01.702 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:56:01.702 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:56:01.702 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131773, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:56:01.699+08:00, endTime=null} 2023-05-15 13:56:01.706 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,0b7c14fc4453] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131769, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.706 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:56:01.706 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:56:01.707 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:56:01.707 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:56:01.707 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:56:01.707 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e92b8ee05308] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131772, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116188, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.689+08:00, sub_task_instance_id=131772}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.707 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e92b8ee05308] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:56:01.719 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,18356d2b94a0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131773, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116189, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:01.699+08:00, sub_task_instance_id=131773}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:01.726 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:56:01.726 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:56:01.727 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:56:01.727 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:56:01.727 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:56:01.727 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:56:01.728 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:56:01.728 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:56:01.734 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:56:01.735 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:56:01.735 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,18356d2b94a0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131773, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.738 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:56:01.742 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.743 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.743 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.743 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.743 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.744 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:01.745 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.746 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:56:01.746 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-1,fb27d19334114ed7,0b300f582753] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-4,fb27d19334114ed7,6f71ff8c2d64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-3,fb27d19334114ed7,1e96b63a6d36] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-2,fb27d19334114ed7,8792c9af4bdb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:56:01.748 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.749 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.749 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.749 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.749 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.750 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.750 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:01.755 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.755 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.755 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:01.755 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:01.756 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:56:01.756 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:56:01.760 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:56:01.760 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:56:01.760 INFO 11 --- [pool-3241653-thread-6,fb27d19334114ed7,ce6f8a09cec8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:56:01.760 INFO 11 --- [pool-3241653-thread-5,fb27d19334114ed7,c368cd5f5f54] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:56:01.761 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e92b8ee05308] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:56:01.766 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e92b8ee05308] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131772, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:01.772 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 13:56:01.786 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:56:02.097 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116184, status:SUCCESS 2023-05-15 13:56:02.099 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116184, status:SUCCESS 2023-05-15 13:56:02.109 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116185, status:SUCCESS 2023-05-15 13:56:02.110 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116185, status:SUCCESS 2023-05-15 13:56:02.121 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116186, status:SUCCESS 2023-05-15 13:56:02.122 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116186, status:SUCCESS 2023-05-15 13:56:02.134 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116187, status:SUCCESS 2023-05-15 13:56:02.135 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116187, status:SUCCESS 2023-05-15 13:56:02.145 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116188, status:SUCCESS 2023-05-15 13:56:02.147 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116188, status:SUCCESS 2023-05-15 13:56:02.157 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116189, status:SUCCESS 2023-05-15 13:56:02.158 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116189, status:SUCCESS 2023-05-15 13:56:02.490 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:02.491 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:04.374 INFO 11 --- [pool-ocp-schedules-9,,d797697a61e2] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:56:04.375 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:56:04.375 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:56:04.375 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:56:04.375 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:56:04.375 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:56:04.382 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:56:04.382 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:56:04.382 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:56:04.382 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:56:04.382 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:56:04.412 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:56:04.414 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:56:04.414 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:56:04.415 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:56:04.414426869+08:00 2023-05-15 13:56:04.415 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:56:04.415 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:56:04.415+08:00 2023-05-15 13:56:04.416 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:56:04.416234459+08:00 2023-05-15 13:56:04.416 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:56:04.416328124+08:00 2023-05-15 13:56:04.416 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:56:04.416+08:00 2023-05-15 13:56:04.416 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:56:04.417 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:56:04.416+08:00 2023-05-15 13:56:04.417 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:56:04.417142754+08:00 2023-05-15 13:56:04.417 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:56:04.417+08:00 2023-05-15 13:56:04.419 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:56:04.418695951+08:00 2023-05-15 13:56:04.419 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:56:04.415+08:00 and status AVAILABLE 2023-05-15 13:56:04.419 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:56:04.419+08:00 2023-05-15 13:56:04.419 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:56:04.416+08:00 and status AVAILABLE 2023-05-15 13:56:04.420 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:56:04.416+08:00 and status AVAILABLE 2023-05-15 13:56:04.421 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:56:04.417+08:00 and status AVAILABLE 2023-05-15 13:56:04.423 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:56:04.419+08:00 and status AVAILABLE 2023-05-15 13:56:04.436 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:56:04.436 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:56:04.437 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:56:04.437 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:56:04.437 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:56:04.453 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130164 0 0 , error= 2023-05-15 13:56:04.459 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130164 0 0 , error= 2023-05-15 13:56:04.461 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130164 0 0 , error= 2023-05-15 13:56:04.473 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:56:04.461+08:00 2023-05-15 13:56:04.473 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:56:04.461+08:00 2023-05-15 13:56:04.484 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130164 0 0 , error= 2023-05-15 13:56:04.485 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:56:04.484+08:00 2023-05-15 13:56:04.485 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:56:04.485+08:00 2023-05-15 13:56:04.490 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:56:04.461+08:00 and status AVAILABLE 2023-05-15 13:56:04.490 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:56:04.484+08:00 and status AVAILABLE 2023-05-15 13:56:04.490 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:56:04.461+08:00 and status AVAILABLE 2023-05-15 13:56:04.490 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:56:04.485+08:00 and status AVAILABLE 2023-05-15 13:56:04.492 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:56:04.493 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:56:04.493 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:56:04.493 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:56:04.506 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:56:04.506 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:56:04.506 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:56:04.506 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:56:04.535 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:56:04.537 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:56:04.537 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:56:04.539 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:56:04.582 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:56:04.588 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:56:04.591 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:56:04.596 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:56:04.605 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:56:04.605 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:56:04.616 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:56:04.623 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:56:04.626 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:56:04.626 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:56:04.635 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:56:04.651 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:56:04.659 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:56:04.672 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:56:04.680 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:56:04.688 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:56:04.692 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:56:04.695 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:56:04.700 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:56:04.709 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:56:04.713 INFO 11 --- [pool-ocp-schedules-12,,170556b97e74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:56:04.720 INFO 11 --- [pool-ocp-schedules-12,,170556b97e74] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:56:04.729 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:56:04.734 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:56:04.748 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:56:04.748 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:56:04.760 INFO 11 --- [pool-Check host status9,,589ccf17de96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:56:04.768 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:56:04.770 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:56:04.779 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:56:04.785 INFO 11 --- [pool-Check host status22,,c86e08f8c586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:56:04.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:04.796 INFO 11 --- [pool-Check host status26,,c249208b06af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:56:04.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:04.829358+08:00, lastProcessTime=2023-05-15T13:55:54.833447+08:00 2023-05-15 13:56:04.849 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:56:04.869 INFO 11 --- [alarm-detect-2,,334bed3f058a] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130144 ~ 1684130154 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.316, values={1684130150=245.316}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:04.873 INFO 11 --- [pool-Check host status10,,74c4d0627534] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:56:04.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:04.829358+08:00 2023-05-15 13:56:05.207 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:05.0, nextRun=2023-05-15 13:56:05.0, currentDate=Mon May 15 13:56:05 CST 2023 2023-05-15 13:56:05.207 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:55:05.0, nextRunTime=2023-05-15 13:56:05.0} 2023-05-15 13:56:05.211 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:56:05.211 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:05.211 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:05.394 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130165 0 0 , error= 2023-05-15 13:56:05.399 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:56:05.399+08:00 2023-05-15 13:56:05.408 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:56:05.399+08:00 and status AVAILABLE 2023-05-15 13:56:05.410 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:56:05.421 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:56:05.451 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:56:05.491 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:05.492 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:05.512 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:56:05.526 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:56:05.558 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:56:05.622 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:56:05.638 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:56:05.696 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:56:05.748 INFO 11 --- [pool-ocp-schedules-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-05-15 13:56:05.755 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:56:05.771 INFO 11 --- [pool-Check host status12,,97fc8be16dcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:56:05.788 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:05.804 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:05.806 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:56:05.806 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:56:05.808 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:56:05.809 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:56:05.809 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:56:05.810 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:05.812 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:56:05.812 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:05.813 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:56:05.816 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:56:05.817 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:56:05.820 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131774, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:05.821 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:56:05.822 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131774, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:56:05.818+08:00, endTime=null} 2023-05-15 13:56:05.841 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:56:05.841 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:56:05.842 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8c913a604878] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131774, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116190, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:05.817+08:00, sub_task_instance_id=131774}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:05.842 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:56:05.844 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:56:05.844 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:56:05.862 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,fe906760baab] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:56:05.862 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status15,,b281af736515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,74a8fb0521ea] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,96c1a02794f0] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status24,,4372fcdeb8d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:56:05.863 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:56:05.864 INFO 11 --- [pool-Check host status25,,73172796d4cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:56:05.865 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:56:05.865 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:56:05.865 INFO 11 --- [pool-Check host status16,,023b7ad2066a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:56:05.866 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:56:05.866 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:56:05.866 INFO 11 --- [pool-Check host status18,,e9e0e06ceebe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:56:05.903 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,96c1a02794f0] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:56:05.913 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,74a8fb0521ea] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:56:05.914 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,fe906760baab] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:56:05.926 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8c913a604878] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131774, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:06.322 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116190, status:SUCCESS 2023-05-15 13:56:06.323 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116190, status:SUCCESS 2023-05-15 13:56:06.781 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:06.899 INFO 11 --- [pool-ocp-schedules-16,,276026bf37e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:08.251 INFO 11 --- [pool-overview-container-2,,5e91be141d5c] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:56:08.252 INFO 11 --- [pool-overview-container-2,,5e91be141d5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:08.492 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:08.494 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:08.499 INFO 11 --- [pool-scanner--15966599-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:56:09.520 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,30d4610daff64f1c,f39178f6f818] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=30d4610daff64f1c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:09.526 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,30d4610daff64f1c,f39178f6f818] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=30d4610daff64f1c, duration=6 ms] 2023-05-15 13:56:09.925 INFO 11 --- [pool-ocp-schedules-18,,3751d5b81f7f] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:56:11.494 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:11.495 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:11.609 INFO 11 --- [pool-ocp-schedules-5,,660fc6077b16] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:11.609 INFO 11 --- [pool-ocp-schedules-5,,660fc6077b16] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:12.216 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:14.142 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130100, nodeCount=1000, affectRows=1000 2023-05-15 13:56:14.152 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130100, nodeCount=1000, validCount=1000 2023-05-15 13:56:14.158 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130100, nodeCount=902, affectRows=902 2023-05-15 13:56:14.158 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130100, nodeCount=788, affectRows=788 2023-05-15 13:56:14.165 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130100, nodeCount=788, validCount=788 2023-05-15 13:56:14.166 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130100, nodeCount=902, validCount=902 2023-05-15 13:56:14.495 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:14.496 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:14.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:14.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:14.831995+08:00, lastProcessTime=2023-05-15T13:56:04.829358+08:00 2023-05-15 13:56:14.860 INFO 11 --- [alarm-detect-9,,2873bd152730] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130154 ~ 1684130164 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.361, values={1684130160=245.361}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:14.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:14.831995+08:00 2023-05-15 13:56:15.720 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,3eb56de61cd84829,973400938cb4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=3eb56de61cd84829, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:15.725 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,3eb56de61cd84829,973400938cb4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=3eb56de61cd84829, duration=5 ms] 2023-05-15 13:56:15.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:15.810 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:15.818 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:15.824 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:16.785 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:16.920 INFO 11 --- [pool-ocp-schedules-17,,6ffec7afbed5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:17.497 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:17.507 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:18.251 INFO 11 --- [pool-overview-container-4,,c44f8a8856de] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:18.503 INFO 11 --- [pool-scanner--15966699-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:19.811 INFO 11 --- [pool-scanner--15966700-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:19.831 INFO 11 --- [pool-scanner--15966701-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:19.852 INFO 11 --- [pool-scanner--15966702-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:19.872 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:19.877 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:19.916 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:19.921 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:19.939 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:19.943 INFO 11 --- [pool-ocp-schedules-4,,513540f6bedd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:20.507 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:20.508 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:20.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:21.0, nextRun=2023-05-15 13:56:20.0, currentDate=Mon May 15 13:56:20 CST 2023 2023-05-15 13:56:20.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:55:21.0, nextRunTime=2023-05-15 13:56:20.0} 2023-05-15 13:56:20.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:56:20.681 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:20.681 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:21.051 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131775, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:21.053 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131775, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:56:21.047+08:00, endTime=null} 2023-05-15 13:56:21.077 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,192222362fce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131775, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116191, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:21.047+08:00, sub_task_instance_id=131775}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:21.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,192222362fce] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:56:21.080 INFO 11 --- [ForkJoinPool-65972-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:56:21.080 INFO 11 --- [ForkJoinPool-65972-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:56:21.085 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,192222362fce] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131775, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:21.627 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116191, status:SUCCESS 2023-05-15 13:56:21.629 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116191, status:SUCCESS 2023-05-15 13:56:21.764 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:21.768 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:21.770 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:22.954 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,db076d22c27f4c82,07f424b61985] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=db076d22c27f4c82, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:56:22.956 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,db076d22c27f4c82,07f424b61985] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=db076d22c27f4c82, duration=2 ms] 2023-05-15 13:56:23.508 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:23.509 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:24.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:24.831870+08:00, lastProcessTime=2023-05-15T13:56:14.831995+08:00 2023-05-15 13:56:24.866 INFO 11 --- [alarm-detect-6,,0487810c6d53] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130164 ~ 1684130174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.406, values={1684130170=245.406}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:24.867 INFO 11 --- [alarm-detect-9,,bf4549d7cb9f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130114 ~ 1684130174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388557.988, values={1684130160=388557.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:56:24.868 INFO 11 --- [alarm-detect-9,,bf4549d7cb9f] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130114 ~ 1684130174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993355.358, values={1684130160=993355.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:24.870 INFO 11 --- [alarm-detect-12,,cd2cad2b96d6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130114 ~ 1684130174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388557.938, values={1684130160=388557.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:56:24.870 INFO 11 --- [alarm-detect-12,,cd2cad2b96d6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130114 ~ 1684130174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993355.107, values={1684130160=993355.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:24.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:24.831870+08:00 2023-05-15 13:56:25.798 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:25.818 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:25.824 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:25.827 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:25.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:26.0, nextRun=2023-05-15 13:56:25.0, currentDate=Mon May 15 13:56:25 CST 2023 2023-05-15 13:56:25.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:55:26.0, nextRunTime=2023-05-15 13:56:25.0} 2023-05-15 13:56:25.853 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:56:25.854 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:25.854 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:26.133 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131776, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:26.135 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131776, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:56:26.130+08:00, endTime=null} 2023-05-15 13:56:26.160 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131776, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116192, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:26.130+08:00, sub_task_instance_id=131776}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:56:26.209 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:56:26.210 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:56:26.211 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:56:26.212 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:56:26.213 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:56:26.219 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6c3d7d984020] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131776, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:26.509 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:26.511 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:26.741 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116192, status:SUCCESS 2023-05-15 13:56:26.742 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116192, status:SUCCESS 2023-05-15 13:56:26.788 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:26.940 INFO 11 --- [pool-ocp-schedules-17,,01f5a49b91f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:27.218 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:28.251 INFO 11 --- [pool-overview-container-1,,e72dd5f4120f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:28.502 INFO 11 --- [pool-scanner--15966803-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:29.122 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,dd64f565b9a743bd,c3327943ade9] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=dd64f565b9a743bd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:29.128 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,dd64f565b9a743bd,c3327943ade9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=dd64f565b9a743bd, duration=6 ms] 2023-05-15 13:56:29.511 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:29.512 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:30.517 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,10d2507e5fd047fa,a546176f8a9e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=10d2507e5fd047fa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:30.523 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,10d2507e5fd047fa,a546176f8a9e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=10d2507e5fd047fa, duration=6 ms] 2023-05-15 13:56:31.014 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:30.0, nextRun=2023-05-15 13:56:30.0, currentDate=Mon May 15 13:56:31 CST 2023 2023-05-15 13:56:31.014 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:55:30.0, nextRunTime=2023-05-15 13:56:30.0} 2023-05-15 13:56:31.019 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:56:31.019 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:31.019 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:31.019 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:31.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:56:30.0, currentDate=Mon May 15 13:56:31 CST 2023 2023-05-15 13:56:31.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:56:30.0} 2023-05-15 13:56:31.034 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:56:31.034 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:56:31.034 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:56:31.263 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131778, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:31.270 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131778, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:56:31.260+08:00, endTime=null} 2023-05-15 13:56:31.286 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131779, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:31.293 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131779, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:56:31.283+08:00, endTime=null} 2023-05-15 13:56:31.314 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131777, operation:EXECUTE, state: RUNNING 2023-05-15 13:56:31.319 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131777, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:56:31.312+08:00, endTime=null} 2023-05-15 13:56:31.323 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,88c0abed94c6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131779, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116194, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:31.283+08:00, sub_task_instance_id=131779}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:56:31.323 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,88c0abed94c6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:56:31.347 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:56:31.347 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:56:31.347 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:56:31.347 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,f69c468069324fcd,7636a52b1858] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=f69c468069324fcd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:31.348 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:56:31.348 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:56:31.350 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:56:31.361 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,f69c468069324fcd,7636a52b1858] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=f69c468069324fcd, duration=14 ms] 2023-05-15 13:56:31.373 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131777, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116193, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:31.311+08:00, sub_task_instance_id=131777}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:56:31.375 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.375 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.375 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.375 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131778, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116193, task_operation=execute, latest_execution_start_time=2023-05-15T13:56:31.254+08:00, sub_task_instance_id=131778}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:56:31.376 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.376 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.377 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-1,fb27d19334114ed7,e274c239f398] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:56:31.379 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.380 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.381 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.381 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:56:31.382 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.382 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.382 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.382 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.382 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.383 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:56:31.383 INFO 11 --- [pool-3241813-thread-2,fb27d19334114ed7,c7279fc09850] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:56:31.383 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.384 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.385 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.385 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.385 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:56:31.386 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:56:31.387 INFO 11 --- [pool-3241813-thread-3,fb27d19334114ed7,1573d416442c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:56:31.387 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:56:31.387 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.388 INFO 11 --- [pool-3241813-thread-4,fb27d19334114ed7,4e9572c9d49b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:56:31.388 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.388 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.389 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.389 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.389 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.390 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:56:31.393 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.393 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.393 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:56:31.394 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:56:31.395 INFO 11 --- [pool-3241813-thread-5,fb27d19334114ed7,462aca23787d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:56:31.398 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:56:31.398 INFO 11 --- [pool-3241813-thread-6,fb27d19334114ed7,f32f563effc2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:56:31.399 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,88c0abed94c6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:56:31.403 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,88c0abed94c6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131779, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:31.406 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.406 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.406 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.407 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.408 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.408 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.408 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:56:31.416 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.428 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.429 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.429 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.430 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.431 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.431 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.431 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.435 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.435 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.436 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:56:31.438 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.438 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.438 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.440 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.440 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.440 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.450 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.450 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.452 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.452 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.452 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.464 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.464 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.464 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:56:31.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.475 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.475 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.475 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:56:31.485 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.485 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.485 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:56:31.491 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.491 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.492 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.498 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.498 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.499 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:56:31.503 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.503 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.504 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.509 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:56:31.523 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:56:31.531 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.532 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.532 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.532 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.533 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.534 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.534 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.535 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.548 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.549 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.549 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.549 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.550 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.551 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.551 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.551 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.555 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.556 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.557 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.557 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.557 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.562 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.562 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.563 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.565 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.565 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.565 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.565 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.565 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.566 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.566 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.567 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.576 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.576 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.576 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.577 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.578 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.578 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.578 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.578 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.585 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.585 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.586 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.596 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.599 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.602 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.602 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.603 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.613 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.613 INFO 11 --- [pool-ocp-schedules-16,,47060a853df5] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:31.613 INFO 11 --- [pool-ocp-schedules-16,,47060a853df5] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:31.613 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.613 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.613 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.614 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.614 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.614 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.619 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.619 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.620 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.630 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.631 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.631 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.631 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.631 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:56:31.635 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.636 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.636 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.637 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.637 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.641 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.641 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.641 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.642 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:56:31.643 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.643 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.643 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:56:31.643 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:56:31.645 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:56:31.645 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:56:31.662 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.662 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.663 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.663 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.663 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.665 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.665 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.665 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.669 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:31.670 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.674 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.676 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:56:31.676 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.676 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.676 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:56:31.677 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.677 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.677 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.686 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.689 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:56:31.689 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.689 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.690 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:56:31.692 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.693 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:56:31.694 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.694 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.695 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:56:31.698 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.700 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:56:31.700 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.700 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.701 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:56:31.709 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.710 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:56:31.711 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:56:31.712 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:56:31.712 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:56:31.712 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:56:31.712 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:56:31.712 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:56:31.715 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c13230aec3cb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131778, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:31.715 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76bb5a41ea01] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131777, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:56:31.841 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116193, status:SUCCESS 2023-05-15 13:56:31.842 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116193, status:SUCCESS 2023-05-15 13:56:31.854 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116194, status:SUCCESS 2023-05-15 13:56:31.855 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116194, status:SUCCESS 2023-05-15 13:56:31.944 INFO 11 --- [pool-ocp-schedules-4,,5f4112b66534] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:56:31.947 INFO 11 --- [pool-ocp-schedules-4,,5f4112b66534] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:56:31.963 INFO 11 --- [pool-ocp-schedules-14,,7f82d98dd8cf] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:56:31.965 INFO 11 --- [pool-ocp-schedules-14,,7f82d98dd8cf] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:56:31.970 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.970 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.970 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.971 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.972 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:31.972 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.972 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.973 INFO 11 --- [ForkJoinPool-65973-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:56:31.973 INFO 11 --- [pool-ocp-schedules-1,,e68c51932c81] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:56:31.976 INFO 11 --- [pool-ocp-schedules-1,,e68c51932c81] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:56:31.976 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.976 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.976 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.977 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.982 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:31.982 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:31.982 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:31.986 INFO 11 --- [ForkJoinPool-65973-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:56:32.007 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.007 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.007 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.008 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.009 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.009 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.009 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:56:32.026 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.027 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.027 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.027 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.032 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.032 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.033 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:56:32.061 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:32.062 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.062 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.062 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.062 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.064 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.064 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.064 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:56:32.065 WARN 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:56:32.066 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.066 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.066 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:56:32.068 INFO 11 --- [ForkJoinPool-65975-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:56:32.068 INFO 11 --- [ForkJoinPool-65975-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:56:32.068 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:56:32.080 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.080 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:56:32.094 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.108 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:32.108 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.108 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.108 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.109 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.110 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:56:32.110 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.110 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.114 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:32.115 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.116 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.116 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.117 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.122 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.122 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.122 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:56:32.122 INFO 11 --- [ForkJoinPool-65975-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:56:32.122 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:56:32.124 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.124 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.125 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:56:32.127 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:56:32.139 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.140 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:56:32.154 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.154 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:56:32.172 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.172 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:56:32.190 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.190 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:56:32.207 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.207 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:56:32.225 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:56:32.245 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:56:32.246 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.246 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.246 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.247 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.251 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:56:32.251 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:56:32.252 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:56:32.273 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:56:32.273 INFO 11 --- [ForkJoinPool-65975-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:56:32.512 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:32.514 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:34.826 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:34.825527+08:00, lastProcessTime=2023-05-15T13:56:24.831870+08:00 2023-05-15 13:56:34.856 INFO 11 --- [alarm-detect-10,,816383d47794] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130174 ~ 1684130184 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.406, values={1684130180=245.406}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:34.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:34.825527+08:00 2023-05-15 13:56:35.514 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:35.515 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:35.801 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:35.820 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:35.825 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:35.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:36.792 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:36.961 INFO 11 --- [pool-ocp-schedules-5,,a22b7863442f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:38.252 INFO 11 --- [pool-overview-container-5,,4cc6b7613a1b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:38.508 INFO 11 --- [pool-scanner--15966911-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:38.515 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:38.516 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:41.517 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:41.518 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:42.221 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:43.423 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,e3fd9387434d4a75,ea5c44eb133f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=e3fd9387434d4a75, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:43.430 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,e3fd9387434d4a75,ea5c44eb133f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=e3fd9387434d4a75, duration=8 ms] 2023-05-15 13:56:44.518 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:44.519 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:44.831751+08:00, lastProcessTime=2023-05-15T13:56:34.825527+08:00 2023-05-15 13:56:44.865 INFO 11 --- [alarm-detect-6,,7e773a5f0857] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130184 ~ 1684130194 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.406, values={1684130190=245.406}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:44.957 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:44.831751+08:00 2023-05-15 13:56:45.449 INFO 11 --- [pool-ocp-schedules-13,,68bf586ea9bf] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:56:45.808 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:45.823 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:45.828 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:45.832 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:46.631 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628} 2023-05-15 13:56:46.659 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.660 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.661 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.661 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.669 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.669 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.670 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.671 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.672 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.673 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.680 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:52:46.568, end=2023-05-15T13:54:46.628},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:56:46.680 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:56:46.680 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 13:56:46.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:46.979 INFO 11 --- [pool-ocp-schedules-10,,85052059abc0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:47.519 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:47.520 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:48.251 INFO 11 --- [pool-overview-container-2,,20e1c6f72415] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:48.503 INFO 11 --- [pool-scanner--15967011-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:48.619 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,252c417b71174e71,4903379f658e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=252c417b71174e71, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:56:48.624 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,252c417b71174e71,4903379f658e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=252c417b71174e71, duration=6 ms] 2023-05-15 13:56:50.520 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:50.521 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:51.618 INFO 11 --- [pool-ocp-schedules-19,,2abcf312f95b] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:51.618 INFO 11 --- [pool-ocp-schedules-19,,2abcf312f95b] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:56:51.766 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:51.772 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:51.775 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,30a97340aabd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40108} 2023-05-15 13:56:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,30a97340aabd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:56:53.522 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:53.523 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:54.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:56:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:56:54.832466+08:00, lastProcessTime=2023-05-15T13:56:44.831751+08:00 2023-05-15 13:56:54.859 INFO 11 --- [alarm-detect-5,,40c4bef4228e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130194 ~ 1684130204 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.451, values={1684130200=245.451}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:56:54.903 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:56:54.832466+08:00 2023-05-15 13:56:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,05edda8c8afa] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:56:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,3b676a70546b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:56:55.811 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:55.825 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:55.830 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:55.834 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:56.523 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:56.524 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:56:56.799 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:56.986 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,d7d0d4fd3bda4300,471e9d9d856c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=d7d0d4fd3bda4300, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:56:56.988 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,d7d0d4fd3bda4300,471e9d9d856c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=d7d0d4fd3bda4300, duration=2 ms] 2023-05-15 13:56:57.000 INFO 11 --- [pool-ocp-schedules-6,,73c0f6bbf605] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:56:57.224 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:56:58.251 INFO 11 --- [pool-overview-container-4,,c65c7fe91957] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:56:58.505 INFO 11 --- [pool-scanner--15967105-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:56:59.524 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:56:59.525 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 13:57:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 13:57:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:57:00.000 INFO 11 --- [pool-ocp-schedules-7,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:57:00.001 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130220 2023-05-15 13:57:00.001 INFO 11 --- [pool-ocp-schedules-7,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:57:00.013 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 13:57:00.013 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 13:57:00.013 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 13:57:00.013 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 13:57:00.019 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 13:57:00.019 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 13:57:00.019 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 13:57:00.019 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 13:57:00.046 INFO 11 --- [pool-ocp-schedules-7,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:57:00.057 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 13:57:00.065 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 13:57:00.065 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 13:57:00.065 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 13:57:00.065 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 13:57:00.065 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 13:57:00.073 WARN 11 --- [pool-ocp-schedules-14,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 13:57:00.073 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 13:57:00.593 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 13:57:00.923 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:57:00.0, currentDate=Mon May 15 13:57:00 CST 2023 2023-05-15 13:57:00.923 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:57:00.0} 2023-05-15 13:57:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:57:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:00.939 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:57:00.0, currentDate=Mon May 15 13:57:00 CST 2023 2023-05-15 13:57:00.940 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:57:00.0} 2023-05-15 13:57:00.951 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:57:00.951 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:00.951 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:00.964 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:57:00.0, currentDate=Mon May 15 13:57:00 CST 2023 2023-05-15 13:57:00.964 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:57:00.0} 2023-05-15 13:57:00.975 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:57:00.975 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:00.976 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:00.989 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:31.0, nextRun=2023-05-15 13:57:00.0, currentDate=Mon May 15 13:57:00 CST 2023 2023-05-15 13:57:00.990 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:56:31.0, nextRunTime=2023-05-15 13:57:00.0} 2023-05-15 13:57:00.993 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:57:00.993 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:00.993 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:01.003 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:57:00.0, currentDate=Mon May 15 13:57:01 CST 2023 2023-05-15 13:57:01.003 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:57:00.0} 2023-05-15 13:57:01.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:57:01.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:01.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:01.205 WARN 11 --- [pool-ocp-schedules-14,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 13:57:01.207 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.187833928S 2023-05-15 13:57:01.207 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 13:57:01.332 INFO 11 --- [pool-ocp-schedules-14,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 13:57:01.391 INFO 11 --- [pool-ocp-schedules-14,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 13:57:01.432 INFO 11 --- [pool-ocp-schedules-14,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 13:57:01.442 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 13:57:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,744554746b8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130280, interval=60 2023-05-15 13:57:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,6d89d897ee73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130280, interval=60 2023-05-15 13:57:01.543 INFO 11 --- [pool-prometheus-minute-schedule-1,,744554746b8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:57:01.543 INFO 11 --- [pool-prometheus-minute-schedule-2,,6d89d897ee73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:57:01.543 INFO 11 --- [pool-prometheus-minute-schedule-1,,744554746b8e] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130221, count=8, elapsed=1682446091322 2023-05-15 13:57:01.543 INFO 11 --- [pool-prometheus-minute-schedule-2,,6d89d897ee73] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130221, count=6, elapsed=1682446091322 2023-05-15 13:57:01.614 INFO 11 --- [pool-ocp-schedules-14,,] o.s.boot.SpringApplication : Started application in 1.6 seconds (JVM running for 599877.587) 2023-05-15 13:57:01.614 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.601007446S 2023-05-15 13:57:01.782 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131782, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:01.784 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131782, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:57:01.775+08:00, endTime=null} 2023-05-15 13:57:01.793 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131783, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:01.794 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131783, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:57:01.791+08:00, endTime=null} 2023-05-15 13:57:01.802 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,20af57c67cf5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131782, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116197, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:01.775+08:00, sub_task_instance_id=131782}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:01.805 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131780, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:01.806 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131780, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:57:01.802+08:00, endTime=null} 2023-05-15 13:57:01.812 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f89df0fc9899] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131783, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116198, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:01.791+08:00, sub_task_instance_id=131783}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:01.812 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f89df0fc9899] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:57:01.819 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131781, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:01.821 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131781, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:57:01.816+08:00, endTime=null} 2023-05-15 13:57:01.826 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,efcc339eaa27] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131780, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116195, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:01.802+08:00, sub_task_instance_id=131780}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:01.826 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,20af57c67cf5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:57:01.826 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,20af57c67cf5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:57:01.829 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,20af57c67cf5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 13:57:01.831 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:57:01.831 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:57:01.831 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:57:01.831 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:57:01.832 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:57:01.832 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131784, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:01.834 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:57:01.834 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:57:01.834 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:57:01.834 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131784, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:57:01.829+08:00, endTime=null} 2023-05-15 13:57:01.834 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:57:01.834 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 13:57:01.834 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:57:01.834 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 13:57:01.834 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 13:57:01.834 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 13:57:01.835 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,20af57c67cf5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131782, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:01.836 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,efcc339eaa27] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131780, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:01.837 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:57:01.838 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:57:01.838 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:57:01.838 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:57:01.838 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:57:01.839 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:57:01.842 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,b2a8f0538c3c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131781, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116196, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:01.816+08:00, sub_task_instance_id=131781}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.848 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.849 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.851 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.853 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.853 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.853 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.854 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.854 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.855 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.855 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.856 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:01.856 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.856 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.856 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:57:01.857 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d0dea4a5a4b4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131784, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116199, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:01.829+08:00, sub_task_instance_id=131784}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:01.859 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:57:01.860 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.860 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.861 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:01.861 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.861 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:57:01.861 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:57:01.862 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,b2a8f0538c3c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:57:01.862 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:57:01.862 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,b2a8f0538c3c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:57:01.863 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.863 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.863 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:57:01.863 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:57:01.863 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:57:01.864 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:57:01.865 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.865 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.865 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:57:01.865 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:57:01.865 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 13:57:01.865 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:57:01.865 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:57:01.866 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:57:01.866 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 13:57:01.866 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:01.866 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:01.866 INFO 11 --- [pool-3241980-thread-4,fb27d19334114ed7,e651513aeef3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:57:01.866 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,b2a8f0538c3c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131781, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:01.866 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:57:01.867 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:57:01.867 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 13:57:01.868 INFO 11 --- [pool-3241980-thread-2,fb27d19334114ed7,a6a0ac640ce1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:57:01.868 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:57:01.869 INFO 11 --- [pool-3241980-thread-3,fb27d19334114ed7,028ea15c532b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:57:01.871 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d0dea4a5a4b4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131784, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:01.873 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:57:01.873 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:57:01.874 INFO 11 --- [pool-3241980-thread-6,fb27d19334114ed7,549e45520f4a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:57:01.875 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:57:01.875 INFO 11 --- [pool-3241980-thread-1,fb27d19334114ed7,4cec893dda46] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:57:01.877 INFO 11 --- [pool-3241980-thread-5,fb27d19334114ed7,6216245133c8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:57:01.877 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f89df0fc9899] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:57:01.880 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:57:01.881 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:57:01.881 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f89df0fc9899] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131783, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:02.083 INFO 11 --- [pool-ocp-schedules-14,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 13:57:02.404 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116195, status:SUCCESS 2023-05-15 13:57:02.405 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116195, status:SUCCESS 2023-05-15 13:57:02.414 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116196, status:SUCCESS 2023-05-15 13:57:02.415 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116196, status:SUCCESS 2023-05-15 13:57:02.423 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116197, status:SUCCESS 2023-05-15 13:57:02.425 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116197, status:SUCCESS 2023-05-15 13:57:02.434 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116198, status:SUCCESS 2023-05-15 13:57:02.435 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116198, status:SUCCESS 2023-05-15 13:57:02.446 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116199, status:SUCCESS 2023-05-15 13:57:02.447 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116199, status:SUCCESS 2023-05-15 13:57:02.525 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:02.526 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:03.021 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,7e27664f97554530,5b43b9b9ff68] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=7e27664f97554530, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:03.027 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,7e27664f97554530,5b43b9b9ff68] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=7e27664f97554530, duration=7 ms] 2023-05-15 13:57:04.713 INFO 11 --- [pool-ocp-schedules-12,,67d48624b02a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:57:04.721 INFO 11 --- [pool-ocp-schedules-12,,67d48624b02a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:57:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:04.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:04.830250+08:00, lastProcessTime=2023-05-15T13:56:54.832466+08:00 2023-05-15 13:57:04.858 INFO 11 --- [alarm-detect-10,,0f5638386db6] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130204 ~ 1684130214 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.496, values={1684130210=245.496}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:04.887 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:04.830250+08:00 2023-05-15 13:57:05.198 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:05.0, nextRun=2023-05-15 13:57:05.0, currentDate=Mon May 15 13:57:05 CST 2023 2023-05-15 13:57:05.198 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:56:05.0, nextRunTime=2023-05-15 13:57:05.0} 2023-05-15 13:57:05.202 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:57:05.202 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:05.202 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:05.527 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:05.530 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:05.814 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:05.827 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:05.832 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:05.836 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:05.883 INFO 11 --- [pool-ocp-schedules-6,,29025e99ced6] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:57:05.883 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:57:05.883 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:57:05.883 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:57:05.883 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:57:05.883 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:57:05.887 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:57:05.889 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:57:05.889 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:57:05.890 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:57:05.890 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:57:05.891 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131785, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:05.893 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131785, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:57:05.888+08:00, endTime=null} 2023-05-15 13:57:05.915 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,30681439fc5d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131785, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116200, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:05.888+08:00, sub_task_instance_id=131785}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:05.916 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:57:05.917 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:57:05.919 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:57:05.918765886+08:00 2023-05-15 13:57:05.919 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:57:05.919+08:00 2023-05-15 13:57:05.919 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:57:05.920 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:57:05.919715663+08:00 2023-05-15 13:57:05.920 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:57:05.920+08:00 2023-05-15 13:57:05.921 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:57:05.921500546+08:00 2023-05-15 13:57:05.922 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:57:05.921+08:00 2023-05-15 13:57:05.922 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:57:05.923 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:57:05.925 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:57:05.924501019+08:00 2023-05-15 13:57:05.925 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:57:05.925+08:00 2023-05-15 13:57:05.925 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:57:05.925166797+08:00 2023-05-15 13:57:05.926 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:57:05.925+08:00 2023-05-15 13:57:05.929 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:57:05.919+08:00 and status AVAILABLE 2023-05-15 13:57:05.929 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:57:05.925+08:00 and status AVAILABLE 2023-05-15 13:57:05.930 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:57:05.925+08:00 and status AVAILABLE 2023-05-15 13:57:05.934 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:57:05.935 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:57:05.935 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,0b51b793768a] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:57:05.935 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:57:05.935 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,38ad48b411b7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:57:05.936 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,8d1fd120112b] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:57:05.937 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:57:05.920+08:00 and status AVAILABLE 2023-05-15 13:57:05.956 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130225 0 0 , error= 2023-05-15 13:57:05.967 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130225 0 0 , error= 2023-05-15 13:57:05.967 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:57:05.967+08:00 2023-05-15 13:57:05.967 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:57:05.967+08:00 2023-05-15 13:57:05.968 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:57:05.921+08:00 and status AVAILABLE 2023-05-15 13:57:05.972 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:57:05.967+08:00 and status AVAILABLE 2023-05-15 13:57:05.972 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:57:05.967+08:00 and status AVAILABLE 2023-05-15 13:57:05.973 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:57:05.975 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:57:05.975 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:57:05.976 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:57:05.992 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130225 0 0 , error= 2023-05-15 13:57:05.993 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:57:05.993+08:00 2023-05-15 13:57:06.009 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130225 0 0 , error= 2023-05-15 13:57:06.009 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:57:06.009+08:00 2023-05-15 13:57:06.010 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:57:05.993+08:00 and status AVAILABLE 2023-05-15 13:57:06.014 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:57:06.009+08:00 and status AVAILABLE 2023-05-15 13:57:06.014 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:57:06.014 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:57:06.017 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:57:06.020 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:57:06.023 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:57:06.029 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:57:06.037 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,0b51b793768a] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:57:06.043 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:57:06.043 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:57:06.049 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:57:06.050 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,38ad48b411b7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:57:06.051 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:57:06.054 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,8d1fd120112b] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:57:06.064 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,30681439fc5d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131785, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:06.097 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:57:06.100 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:57:06.101 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:57:06.112 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:57:06.115 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:57:06.115 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:57:06.137 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:57:06.143 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:57:06.149 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:57:06.151 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:57:06.154 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:57:06.171 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:57:06.194 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:57:06.202 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:57:06.205 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:57:06.210 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:57:06.215 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:57:06.216 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:57:06.228 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:57:06.233 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:57:06.234 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:57:06.260 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:57:06.270 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:57:06.271 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:57:06.283 INFO 11 --- [pool-Check host status6,,d12b3de9ff6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:57:06.288 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:57:06.290 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:57:06.294 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:57:06.301 INFO 11 --- [pool-Check host status21,,4f1146295f37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:57:06.304 INFO 11 --- [pool-Check host status4,,87879497c487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:57:06.375 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:57:06.390 INFO 11 --- [pool-Check host status7,,e584523420c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:57:06.536 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116200, status:SUCCESS 2023-05-15 13:57:06.537 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116200, status:SUCCESS 2023-05-15 13:57:06.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:07.021 INFO 11 --- [pool-ocp-schedules-4,,0f6c84732492] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:07.738 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130227 0 0 , error= 2023-05-15 13:57:07.739 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:57:07.738+08:00 2023-05-15 13:57:07.747 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:57:07.738+08:00 and status AVAILABLE 2023-05-15 13:57:07.753 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:57:07.766 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:57:07.795 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:57:07.849 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:57:07.861 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:57:07.875 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:57:07.929 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:57:07.942 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:57:07.962 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:57:08.014 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:57:08.026 INFO 11 --- [pool-Check host status29,,693346bd2e28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:57:08.046 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:57:08.046 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:57:08.046 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:57:08.046 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:57:08.046 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:57:08.051 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:57:08.051 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:57:08.052 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:57:08.053 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:57:08.053 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:57:08.080 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:57:08.082 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:57:08.084 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:57:08.084 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:57:08.084 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:57:08.101 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:57:08.101 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:57:08.101 INFO 11 --- [pool-Check host status23,,a51a30bd1573] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status5,,9e7a7b40a8d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:57:08.104 INFO 11 --- [pool-Check host status20,,338c4da2c428] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status11,,cbb277c3f653] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:57:08.106 INFO 11 --- [pool-Check host status27,,b0d230cd3fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:57:08.254 INFO 11 --- [pool-overview-container-1,,2265975cb442] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 13:57:08.254 INFO 11 --- [pool-overview-container-1,,2265975cb442] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:08.510 INFO 11 --- [pool-scanner--15967198-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:57:08.531 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:08.532 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:09.942 INFO 11 --- [pool-ocp-schedules-15,,7f9ef8394e8a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40109} 2023-05-15 13:57:09.943 INFO 11 --- [pool-ocp-schedules-15,,7f9ef8394e8a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:57:09.954 INFO 11 --- [pool-ocp-schedules-11,,c55f1b27fe53] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 13:57:11.532 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:11.533 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:11.623 INFO 11 --- [pool-ocp-schedules-12,,40035b4f8af6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:11.623 INFO 11 --- [pool-ocp-schedules-12,,40035b4f8af6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:12.227 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:12.444 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,5e9b7797dc0a43de,eae78fbf1645] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=5e9b7797dc0a43de, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:12.450 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,5e9b7797dc0a43de,eae78fbf1645] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=5e9b7797dc0a43de, duration=6 ms] 2023-05-15 13:57:14.174 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130160, nodeCount=690, affectRows=690 2023-05-15 13:57:14.177 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130160, nodeCount=690, validCount=690 2023-05-15 13:57:14.178 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130160, nodeCount=1000, affectRows=1000 2023-05-15 13:57:14.182 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130160, nodeCount=1000, validCount=1000 2023-05-15 13:57:14.182 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130160, nodeCount=1000, affectRows=1000 2023-05-15 13:57:14.188 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130160, nodeCount=1000, validCount=1000 2023-05-15 13:57:14.534 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:14.535 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:14.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:14.831190+08:00, lastProcessTime=2023-05-15T13:57:04.830250+08:00 2023-05-15 13:57:14.868 INFO 11 --- [alarm-detect-5,,f1228042a656] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130214 ~ 1684130224 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130220=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:14.869 INFO 11 --- [alarm-detect-5,,f1228042a656] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 13:57:14.901 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:14.831190+08:00 2023-05-15 13:57:15.816 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:15.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:15.834 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:15.838 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:16.805 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:17.043 INFO 11 --- [pool-ocp-schedules-12,,6fc2a60106a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:17.338 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,57f3b1eb429742fd,56bc03d0ba8d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=57f3b1eb429742fd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:17.343 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,57f3b1eb429742fd,56bc03d0ba8d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=57f3b1eb429742fd, duration=6 ms] 2023-05-15 13:57:17.535 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:17.536 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:17.614 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 13:57:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:17.676 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:17.676 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.677 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:57:17.698 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:17.699 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:17.700 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:17.700 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.701 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:17.705 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:17.705 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.706 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:57:17.709 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 13:57:17.709 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 13:57:17.709 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 13:57:17.709 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 13:57:17.715 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:57:17.715 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.715 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:57:17.716 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.716 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:57:17.716 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:17.716 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:57:17.716 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:57:17.718 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:57:17.737 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T13:55:02, start=2023-05-15T13:50:47, end=2023-05-15T13:53:17, updates=99, granule=120000ms, delay=255000ms 2023-05-15 13:57:17.739 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:55:02, target=2023-05-15T13:53:17, granule=120000ms, delay=105000ms) 2023-05-15 13:57:17.743 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T13:54:48, start=2023-05-15T13:50:47, end=2023-05-15T13:53:17, updates=89, granule=120000ms, delay=241000ms 2023-05-15 13:57:17.745 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:54:48, target=2023-05-15T13:53:17, granule=120000ms, delay=91000ms) 2023-05-15 13:57:17.746 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T13:54:48, start=2023-05-15T13:50:47, end=2023-05-15T13:53:17, updates=125, granule=120000ms, delay=241000ms 2023-05-15 13:57:17.747 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:54:48, target=2023-05-15T13:53:17, granule=120000ms, delay=91000ms) 2023-05-15 13:57:17.828 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=1->2, c=0) - source=2023-05-15T13:53:17, start=2023-05-15T13:39:17, end=2023-05-15T13:51:17, updates=128, granule=600000ms, delay=840000ms 2023-05-15 13:57:17.828 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=1->2, c=0) - source=2023-05-15T13:53:17, start=2023-05-15T13:41:17, end=2023-05-15T13:53:17, updates=128, granule=600000ms, delay=720000ms 2023-05-15 13:57:17.836 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=1) - not enough cached interval (source=2023-05-15T13:53:17, target=2023-05-15T13:51:17, granule=600000ms, delay=120000ms) 2023-05-15 13:57:17.839 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=1) - not enough cached interval (source=2023-05-15T13:53:17, target=2023-05-15T13:53:17, granule=600000ms, delay=0ms) 2023-05-15 13:57:17.840 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=1->2, c=0) - source=2023-05-15T13:53:17, start=2023-05-15T13:39:17, end=2023-05-15T13:51:17, updates=142, granule=600000ms, delay=840000ms 2023-05-15 13:57:17.842 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=1) - not enough cached interval (source=2023-05-15T13:53:17, target=2023-05-15T13:51:17, granule=600000ms, delay=120000ms) 2023-05-15 13:57:17.842 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=228ms, period=120000ms 2023-05-15 13:57:17.889 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,affd34c8a82b4881,f8d25f186fbc] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=affd34c8a82b4881, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:17.894 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,affd34c8a82b4881,f8d25f186fbc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=affd34c8a82b4881, duration=6 ms] 2023-05-15 13:57:18.252 INFO 11 --- [pool-overview-container-5,,69003f039785] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:57:18.253 INFO 11 --- [pool-overview-container-5,,69003f039785] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:18.505 INFO 11 --- [pool-scanner--15967298-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:19.960 INFO 11 --- [pool-ocp-schedules-6,,224180c75025] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 13:57:19.977 INFO 11 --- [pool-scanner--15967299-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:20.001 INFO 11 --- [pool-scanner--15967300-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:20.020 INFO 11 --- [pool-scanner--15967301-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:20.043 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.049 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.094 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.099 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.119 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.123 INFO 11 --- [pool-ocp-schedules-20,,765ca4a9e8e7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:20.536 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:20.537 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:20.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:21.0, nextRun=2023-05-15 13:57:20.0, currentDate=Mon May 15 13:57:20 CST 2023 2023-05-15 13:57:20.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:56:21.0, nextRunTime=2023-05-15 13:57:20.0} 2023-05-15 13:57:20.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:57:20.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:20.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:21.108 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131786, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:21.110 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131786, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:57:21.106+08:00, endTime=null} 2023-05-15 13:57:21.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eca69cb96579] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131786, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116201, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:21.105+08:00, sub_task_instance_id=131786}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:21.138 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eca69cb96579] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:57:21.139 INFO 11 --- [ForkJoinPool-65978-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:57:21.139 INFO 11 --- [ForkJoinPool-65978-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:57:21.145 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eca69cb96579] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131786, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:21.772 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:21.775 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:21.778 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:21.825 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116201, status:SUCCESS 2023-05-15 13:57:21.827 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116201, status:SUCCESS 2023-05-15 13:57:23.538 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:23.539 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:24.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:24.831982+08:00, lastProcessTime=2023-05-15T13:57:14.831190+08:00 2023-05-15 13:57:24.858 INFO 11 --- [alarm-detect-10,,673c1b225667] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130224 ~ 1684130234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130230=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:24.860 INFO 11 --- [alarm-detect-9,,4c65962e3969] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130174 ~ 1684130234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388617.988, values={1684130220=388617.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:57:24.861 INFO 11 --- [alarm-detect-9,,4c65962e3969] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130174 ~ 1684130234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993415.358, values={1684130220=993415.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:24.863 INFO 11 --- [alarm-detect-9,,4c65962e3969] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 13:57:24.863 INFO 11 --- [alarm-detect-6,,eedfea858522] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130174 ~ 1684130234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388617.938, values={1684130220=388617.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:57:24.864 INFO 11 --- [alarm-detect-6,,eedfea858522] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130174 ~ 1684130234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993415.107, values={1684130220=993415.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:24.866 INFO 11 --- [alarm-detect-6,,eedfea858522] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 13:57:24.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:24.831982+08:00 2023-05-15 13:57:25.819 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:25.831 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:26.0, nextRun=2023-05-15 13:57:25.0, currentDate=Mon May 15 13:57:25 CST 2023 2023-05-15 13:57:25.831 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:56:26.0, nextRunTime=2023-05-15 13:57:25.0} 2023-05-15 13:57:25.831 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:25.837 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:25.840 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:25.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:57:25.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:25.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:26.194 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131787, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:26.200 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131787, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:57:26.191+08:00, endTime=null} 2023-05-15 13:57:26.227 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131787, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116202, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:26.191+08:00, sub_task_instance_id=131787}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:57:26.272 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:57:26.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:57:26.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:57:26.274 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:57:26.275 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:57:26.280 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6e62956ad722] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131787, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:26.539 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:26.540 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:26.808 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:26.943 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116202, status:SUCCESS 2023-05-15 13:57:26.944 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116202, status:SUCCESS 2023-05-15 13:57:27.059 INFO 11 --- [pool-ocp-schedules-9,,4ce3fcaa4afc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:27.229 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:28.077 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,a4db4afdccb5405b,5cb1b61e4ac4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=a4db4afdccb5405b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:28.083 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,a4db4afdccb5405b,5cb1b61e4ac4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=a4db4afdccb5405b, duration=6 ms] 2023-05-15 13:57:28.254 INFO 11 --- [pool-overview-container-2,,8904dd59e21c] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=4 2023-05-15 13:57:28.254 INFO 11 --- [pool-overview-container-2,,8904dd59e21c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:28.502 INFO 11 --- [pool-scanner--15967402-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:29.540 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:29.542 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:29.967 INFO 11 --- [pool-ocp-schedules-3,,0ec705bcdbad] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=4 2023-05-15 13:57:31.013 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:31.0, nextRun=2023-05-15 13:57:30.0, currentDate=Mon May 15 13:57:31 CST 2023 2023-05-15 13:57:31.013 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:56:31.0, nextRunTime=2023-05-15 13:57:30.0} 2023-05-15 13:57:31.019 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,49ad8a2e8f23470b,b63aaff9a437] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=49ad8a2e8f23470b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:57:31.021 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,49ad8a2e8f23470b,b63aaff9a437] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=49ad8a2e8f23470b, duration=2 ms] 2023-05-15 13:57:31.022 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:57:31.025 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:31.025 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:31.025 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:31.038 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:01.0, nextRun=2023-05-15 13:57:30.0, currentDate=Mon May 15 13:57:31 CST 2023 2023-05-15 13:57:31.039 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:57:01.0, nextRunTime=2023-05-15 13:57:30.0} 2023-05-15 13:57:31.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:57:31.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:57:31.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:57:31.284 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131790, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:31.286 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131790, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:57:31.282+08:00, endTime=null} 2023-05-15 13:57:31.297 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131788, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:31.303 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131788, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:57:31.294+08:00, endTime=null} 2023-05-15 13:57:31.309 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3ac01e221fcb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131790, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116204, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:31.282+08:00, sub_task_instance_id=131790}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:57:31.309 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3ac01e221fcb] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:57:31.317 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131789, operation:EXECUTE, state: RUNNING 2023-05-15 13:57:31.319 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131789, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:57:31.314+08:00, endTime=null} 2023-05-15 13:57:31.328 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131788, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116203, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:31.294+08:00, sub_task_instance_id=131788}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:57:31.332 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:57:31.333 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:57:31.333 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:57:31.333 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:57:31.333 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:57:31.333 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:57:31.345 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131789, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116203, task_operation=execute, latest_execution_start_time=2023-05-15T13:57:31.314+08:00, sub_task_instance_id=131789}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:57:31.345 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.345 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.345 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.346 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.347 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.348 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:57:31.349 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.350 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.350 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:57:31.351 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:57:31.351 INFO 11 --- [pool-3242140-thread-4,fb27d19334114ed7,3d239982d84b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:57:31.351 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:57:31.352 INFO 11 --- [pool-3242140-thread-3,fb27d19334114ed7,a68771abd1d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:57:31.352 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:57:31.352 INFO 11 --- [pool-3242140-thread-1,fb27d19334114ed7,11509ea26f8b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:57:31.353 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.354 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.354 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.354 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:57:31.354 INFO 11 --- [pool-3242140-thread-2,fb27d19334114ed7,f7f365e1812d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:57:31.355 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.355 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.356 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.356 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.356 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.360 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.360 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.360 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.360 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.361 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:57:31.361 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:57:31.366 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:57:31.366 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:57:31.366 INFO 11 --- [pool-3242140-thread-5,fb27d19334114ed7,cb540c4c7c5b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:57:31.366 INFO 11 --- [pool-3242140-thread-6,fb27d19334114ed7,900d59651705] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:57:31.367 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3ac01e221fcb] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:57:31.371 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3ac01e221fcb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131790, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:31.374 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.375 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.375 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.375 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.383 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.383 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.384 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:57:31.394 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.394 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.394 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.405 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.416 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.416 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.416 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.417 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.422 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.422 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.422 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:57:31.423 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.423 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.423 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.428 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.428 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.429 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.433 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.433 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.433 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:57:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:57:31.448 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.448 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.449 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.454 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.454 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.454 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:57:31.462 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.462 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.463 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.465 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.465 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.465 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:57:31.475 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:57:31.481 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.481 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.482 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.492 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:57:31.496 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.497 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.497 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.497 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.497 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.504 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.505 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.505 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.518 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.519 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.519 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.519 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.519 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.520 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.521 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.521 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.530 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.531 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.532 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.532 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.533 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.537 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.537 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.538 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.540 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.540 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.541 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.541 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.541 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.542 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.542 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.543 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.557 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.557 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.557 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.557 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.557 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.558 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.558 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.558 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.558 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.559 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.559 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.559 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.559 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.564 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.564 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.565 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.581 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.581 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.581 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.582 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.582 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.583 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.583 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.584 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.589 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.590 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.590 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.590 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.591 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.594 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.594 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.595 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.595 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.595 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.595 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.595 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.596 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.612 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.613 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.614 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.614 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.615 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.615 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.616 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.616 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.616 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.617 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.618 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.618 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.618 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:57:31.619 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.620 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.620 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.621 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:57:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.632 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.633 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.633 INFO 11 --- [pool-ocp-schedules-12,,3235bf0ae16d] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:31.633 INFO 11 --- [pool-ocp-schedules-12,,3235bf0ae16d] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:31.634 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.634 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.635 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:57:31.635 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:57:31.635 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:57:31.638 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:57:31.660 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.661 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.661 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.661 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.662 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.664 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.665 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.665 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.667 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:31.668 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.669 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.669 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.670 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.675 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.676 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.676 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.681 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.683 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:57:31.684 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.684 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.684 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:57:31.692 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.694 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.694 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:57:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:57:31.696 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:57:31.696 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.696 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.697 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:57:31.706 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.708 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.708 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:57:31.709 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:57:31.709 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:57:31.710 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:57:31.711 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.711 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.711 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:57:31.714 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c599fd3ebca4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131789, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:31.720 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:57:31.723 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:57:31.723 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:57:31.723 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:57:31.728 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ca8fc0508fb2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131788, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:57:31.953 INFO 11 --- [pool-ocp-schedules-15,,389ddd4801b9] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:57:31.955 INFO 11 --- [pool-ocp-schedules-15,,389ddd4801b9] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:57:31.970 INFO 11 --- [pool-ocp-schedules-10,,15b73f421f84] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:57:31.972 INFO 11 --- [pool-ocp-schedules-10,,15b73f421f84] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:57:31.977 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.977 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.977 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.978 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.979 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:31.979 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.979 INFO 11 --- [pool-ocp-schedules-4,,f4267732c8db] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:57:31.980 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.980 INFO 11 --- [ForkJoinPool-65979-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:57:31.982 INFO 11 --- [pool-ocp-schedules-4,,f4267732c8db] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:57:31.982 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.983 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.983 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.984 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.988 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:31.988 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:31.989 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:31.993 INFO 11 --- [ForkJoinPool-65979-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:57:32.012 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.012 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.012 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.012 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.023 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.023 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.024 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:57:32.030 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.030 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.030 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.031 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.033 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116203, status:SUCCESS 2023-05-15 13:57:32.034 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116203, status:SUCCESS 2023-05-15 13:57:32.036 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.036 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.036 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:57:32.047 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116204, status:SUCCESS 2023-05-15 13:57:32.048 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116204, status:SUCCESS 2023-05-15 13:57:32.059 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:32.059 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.059 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.059 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.060 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.061 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.061 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.062 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:57:32.063 WARN 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:57:32.063 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.063 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.063 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:57:32.064 INFO 11 --- [ForkJoinPool-65981-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:57:32.064 INFO 11 --- [ForkJoinPool-65981-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:57:32.065 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:57:32.080 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.080 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:57:32.094 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.103 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.104 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.105 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.105 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.105 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:57:32.106 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.106 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.109 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.109 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.114 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:57:32.116 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:57:32.116 INFO 11 --- [ForkJoinPool-65981-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:57:32.118 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.118 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.118 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:57:32.121 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:57:32.138 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.138 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:57:32.156 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.156 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:57:32.173 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.173 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:57:32.186 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.186 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:57:32.198 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.198 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:57:32.215 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:57:32.234 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:57:32.236 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.236 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.236 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.237 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.241 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:57:32.242 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:57:32.242 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:57:32.264 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:57:32.264 INFO 11 --- [ForkJoinPool-65981-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:57:32.542 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:32.543 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:34.826 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:34.825107+08:00, lastProcessTime=2023-05-15T13:57:24.831982+08:00 2023-05-15 13:57:34.869 INFO 11 --- [alarm-detect-8,,683caabcc2f2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130234 ~ 1684130244 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130240=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:34.908 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:34.825107+08:00 2023-05-15 13:57:35.543 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:35.545 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:35.822 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:35.837 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:35.840 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:35.842 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:35.991 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,5efe28f6f17f47ab,6722e4e4a204] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=5efe28f6f17f47ab, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:35.996 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,5efe28f6f17f47ab,6722e4e4a204] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=5efe28f6f17f47ab, duration=5 ms] 2023-05-15 13:57:36.811 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:36.921 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,22ed61434b024cdd,bc0f466f5a42] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=22ed61434b024cdd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:36.927 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,22ed61434b024cdd,bc0f466f5a42] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=22ed61434b024cdd, duration=6 ms] 2023-05-15 13:57:37.081 INFO 11 --- [pool-ocp-schedules-15,,77552467c556] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:38.252 INFO 11 --- [pool-overview-container-4,,d90d4c4cb19c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:38.501 INFO 11 --- [pool-scanner--15967510-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:38.545 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:38.546 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:41.546 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:41.547 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:42.231 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:44.550 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:44.553 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:44.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:44.830008+08:00, lastProcessTime=2023-05-15T13:57:34.825107+08:00 2023-05-15 13:57:44.871 INFO 11 --- [alarm-detect-15,,0a9b1a034976] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130244 ~ 1684130254 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130250=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:44.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:44.830008+08:00 2023-05-15 13:57:45.451 INFO 11 --- [pool-ocp-schedules-15,,81b11c4d0010] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:57:45.825 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:45.839 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:45.842 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:45.852 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:46.815 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:47.103 INFO 11 --- [pool-ocp-schedules-20,,19dd6e99d96d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:47.554 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:47.554 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:48.253 INFO 11 --- [pool-overview-container-8,,1d71e522ab5b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:48.506 INFO 11 --- [pool-scanner--15967610-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:50.555 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:50.556 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:51.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ae9ff4be66ed48a7,28009cb4492a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=ae9ff4be66ed48a7, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:57:51.244 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ae9ff4be66ed48a7,28009cb4492a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=ae9ff4be66ed48a7, duration=7 ms] 2023-05-15 13:57:51.639 INFO 11 --- [pool-ocp-schedules-19,,a9a05b7a5fae] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:51.639 INFO 11 --- [pool-ocp-schedules-19,,a9a05b7a5fae] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:57:51.774 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:51.778 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:51.780 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,960931614dc2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40109} 2023-05-15 13:57:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,960931614dc2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:57:53.556 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:53.558 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:57:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:57:54.831097+08:00, lastProcessTime=2023-05-15T13:57:44.830008+08:00 2023-05-15 13:57:54.873 INFO 11 --- [alarm-detect-7,,55671d23f575] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130254 ~ 1684130264 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130260=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:57:54.912 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:57:54.831097+08:00 2023-05-15 13:57:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,23fe604f57d1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:57:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,1d7a825eb0e1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:57:55.827 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:55.841 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:55.852 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:55.854 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:56.558 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:56.559 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:57:56.817 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:57.127 INFO 11 --- [pool-ocp-schedules-12,,eb686a6cbc22] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:57:57.235 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:57:58.260 INFO 11 --- [pool-overview-container-6,,73de841297af] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:57:58.503 INFO 11 --- [pool-scanner--15967704-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:57:59.559 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:57:59.560 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:58:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:58:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:58:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130280 2023-05-15 13:58:00.043 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:58:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:56:01.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 13:56:01.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 13:58:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:00.938 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:01.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.938 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:57:01.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.941 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:58:00.941 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.941 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:00.948 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:01.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.948 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:57:01.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:58:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:00.961 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:01.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.961 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:57:01.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.969 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:58:00.969 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.969 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:00.973 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:31.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.974 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:57:31.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.977 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:58:00.977 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.977 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:00.982 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:01.0, nextRun=2023-05-15 13:58:00.0, currentDate=Mon May 15 13:58:00 CST 2023 2023-05-15 13:58:00.982 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:57:01.0, nextRunTime=2023-05-15 13:58:00.0} 2023-05-15 13:58:00.985 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:58:00.985 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:00.985 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,e77ae9ab919b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130340, interval=60 2023-05-15 13:58:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,95db99cc6984] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130340, interval=60 2023-05-15 13:58:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,e77ae9ab919b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:58:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,e77ae9ab919b] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130281, count=8, elapsed=1682446151241 2023-05-15 13:58:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,95db99cc6984] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:58:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,95db99cc6984] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130281, count=6, elapsed=1682446151242 2023-05-15 13:58:01.765 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131794, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.767 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131794, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:58:01.762+08:00, endTime=null} 2023-05-15 13:58:01.780 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131795, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.782 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131795, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:58:01.777+08:00, endTime=null} 2023-05-15 13:58:01.789 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e3cf2d67736d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131794, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116207, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.762+08:00, sub_task_instance_id=131794}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.794 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131792, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.797 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131792, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T13:58:01.792+08:00, endTime=null} 2023-05-15 13:58:01.802 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,746755a7949b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131795, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116208, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.776+08:00, sub_task_instance_id=131795}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.808 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131793, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.809 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131793, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:58:01.805+08:00, endTime=null} 2023-05-15 13:58:01.816 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e3cf2d67736d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:58:01.817 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e3cf2d67736d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:58:01.820 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d7bc8bbf9b14] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131792, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116205, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.792+08:00, sub_task_instance_id=131792}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.820 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131796, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.821 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e3cf2d67736d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131794, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.822 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131796, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:58:01.817+08:00, endTime=null} 2023-05-15 13:58:01.825 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,746755a7949b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:58:01.825 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,746755a7949b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:58:01.829 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,746755a7949b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 13:58:01.829 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,013f3a6b4f94] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131793, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116206, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.805+08:00, sub_task_instance_id=131793}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.833 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131797, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.834 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,746755a7949b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131795, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.835 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131797, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:58:01.830+08:00, endTime=null} 2023-05-15 13:58:01.835 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d7bc8bbf9b14] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131792, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.838 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:58:01.838 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:58:01.838 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:58:01.838 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:58:01.838 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:58:01.841 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,65e15f4faa96] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131796, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116209, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.817+08:00, sub_task_instance_id=131796}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.841 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,65e15f4faa96] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:58:01.842 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,013f3a6b4f94] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131793, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.843 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:58:01.843 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:58:01.843 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:58:01.844 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:58:01.844 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:58:01.847 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131791, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:01.849 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131791, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T13:58:01.845+08:00, endTime=null} 2023-05-15 13:58:01.851 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae691c8dff7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131797, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116210, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.830+08:00, sub_task_instance_id=131797}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:58:01.860 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:58:01.866 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:58:01.867 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:58:01.867 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae691c8dff7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131797, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.868 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:58:01.870 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:58:01.876 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,cac5b3538e72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131791, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116205, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:01.845+08:00, sub_task_instance_id=131791}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:58:01.877 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.877 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.877 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.877 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:58:01.877 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.878 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.881 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.882 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:58:01.883 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:58:01.884 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:01.884 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.884 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:58:01.884 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.885 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.885 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:58:01.885 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.885 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.885 INFO 11 --- [pool-3242307-thread-4,fb27d19334114ed7,bd830c777bb9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-2,fb27d19334114ed7,2171715a4ff3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.886 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:01.887 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:58:01.887 INFO 11 --- [pool-3242307-thread-3,fb27d19334114ed7,b1481fcffc41] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:58:01.887 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:58:01.887 INFO 11 --- [pool-3242307-thread-1,fb27d19334114ed7,c80eb3950606] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:58:01.892 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.892 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.892 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:01.892 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:01.892 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:58:01.893 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:58:01.895 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,cac5b3538e72] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131791, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:01.898 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:58:01.898 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:58:01.898 INFO 11 --- [pool-3242307-thread-6,fb27d19334114ed7,e34c18b77210] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:58:01.898 INFO 11 --- [pool-3242307-thread-5,fb27d19334114ed7,b84d645a3354] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:58:01.899 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,65e15f4faa96] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:58:01.903 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,65e15f4faa96] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131796, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:02.561 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:02.561 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:02.657 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116205, status:SUCCESS 2023-05-15 13:58:02.658 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116205, status:SUCCESS 2023-05-15 13:58:02.669 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116206, status:SUCCESS 2023-05-15 13:58:02.670 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116206, status:SUCCESS 2023-05-15 13:58:02.681 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116207, status:SUCCESS 2023-05-15 13:58:02.683 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116207, status:SUCCESS 2023-05-15 13:58:02.693 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116208, status:SUCCESS 2023-05-15 13:58:02.694 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116208, status:SUCCESS 2023-05-15 13:58:02.704 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116209, status:SUCCESS 2023-05-15 13:58:02.706 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116209, status:SUCCESS 2023-05-15 13:58:02.716 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116210, status:SUCCESS 2023-05-15 13:58:02.717 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116210, status:SUCCESS 2023-05-15 13:58:04.714 INFO 11 --- [pool-ocp-schedules-1,,c53851541d4f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:58:04.722 INFO 11 --- [pool-ocp-schedules-1,,c53851541d4f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:58:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:04.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:04.832608+08:00, lastProcessTime=2023-05-15T13:57:54.831097+08:00 2023-05-15 13:58:04.865 INFO 11 --- [alarm-detect-12,,5a4dc964e5e5] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130264 ~ 1684130274 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130270=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:04.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:04.832608+08:00 2023-05-15 13:58:05.049 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,33c4f030615246b2,92874b3f7a3f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=33c4f030615246b2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:58:05.051 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,33c4f030615246b2,92874b3f7a3f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=33c4f030615246b2, duration=2 ms] 2023-05-15 13:58:05.220 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:05.0, nextRun=2023-05-15 13:58:05.0, currentDate=Mon May 15 13:58:05 CST 2023 2023-05-15 13:58:05.221 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:57:05.0, nextRunTime=2023-05-15 13:58:05.0} 2023-05-15 13:58:05.224 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:58:05.224 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:05.224 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:05.259 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,bd289914b2544421,0e5cf5cffd68] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=bd289914b2544421, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:05.263 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,bd289914b2544421,0e5cf5cffd68] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=bd289914b2544421, duration=4 ms] 2023-05-15 13:58:05.562 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:05.563 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:05.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:05.843 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:05.854 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:05.856 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:05.913 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131798, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:05.915 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131798, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:58:05.910+08:00, endTime=null} 2023-05-15 13:58:05.938 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,32ea1320f318] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131798, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116211, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:05.910+08:00, sub_task_instance_id=131798}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:05.964 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,284f0ec7f58a] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:58:05.964 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,b0c163bec2e5] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:58:05.964 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,1053ab28d5a6] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:58:06.001 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,b0c163bec2e5] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:58:06.004 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,1053ab28d5a6] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:58:06.007 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,284f0ec7f58a] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:58:06.019 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,32ea1320f318] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131798, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:06.815 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116211, status:SUCCESS 2023-05-15 13:58:06.817 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116211, status:SUCCESS 2023-05-15 13:58:06.820 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:07.151 INFO 11 --- [pool-ocp-schedules-4,,145fe6f30d5e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:08.124 INFO 11 --- [pool-ocp-schedules-18,,75fe36dd551b] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:58:08.124 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:58:08.124 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:58:08.124 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:58:08.124 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:58:08.124 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:58:08.130 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:58:08.132 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:58:08.133 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:58:08.134 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:58:08.136 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:58:08.160 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:58:08.160 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:58:08.161 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:58:08.161 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:58:08.162 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:58:08.161828157+08:00 2023-05-15 13:58:08.162 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:58:08.162+08:00 2023-05-15 13:58:08.162 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:58:08.162147475+08:00 2023-05-15 13:58:08.162 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:58:08.162+08:00 2023-05-15 13:58:08.163 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:58:08.163095732+08:00 2023-05-15 13:58:08.163 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:58:08.163+08:00 2023-05-15 13:58:08.164 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:58:08.163504110+08:00 2023-05-15 13:58:08.164 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:58:08.164+08:00 2023-05-15 13:58:08.164 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:58:08.165 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:58:08.162+08:00 and status AVAILABLE 2023-05-15 13:58:08.165 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:58:08.162+08:00 and status AVAILABLE 2023-05-15 13:58:08.167 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:58:08.166615987+08:00 2023-05-15 13:58:08.167 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:58:08.163+08:00 and status AVAILABLE 2023-05-15 13:58:08.167 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:58:08.167+08:00 2023-05-15 13:58:08.168 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:58:08.164+08:00 and status AVAILABLE 2023-05-15 13:58:08.171 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:58:08.167+08:00 and status AVAILABLE 2023-05-15 13:58:08.172 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:58:08.172 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:58:08.173 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:58:08.174 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:58:08.185 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130288 0 0 , error= 2023-05-15 13:58:08.185 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130288 0 0 , error= 2023-05-15 13:58:08.185 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130288 0 0 , error= 2023-05-15 13:58:08.188 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130288 0 0 , error= 2023-05-15 13:58:08.190 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:58:08.192 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:58:08.190+08:00 2023-05-15 13:58:08.192 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:58:08.192+08:00 2023-05-15 13:58:08.192 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:58:08.190+08:00 2023-05-15 13:58:08.192 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:58:08.192+08:00 2023-05-15 13:58:08.211 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130288 0 0 , error= 2023-05-15 13:58:08.211 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:58:08.190+08:00 and status AVAILABLE 2023-05-15 13:58:08.211 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:58:08.211+08:00 2023-05-15 13:58:08.211 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:58:08.190+08:00 and status AVAILABLE 2023-05-15 13:58:08.211 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:58:08.192+08:00 and status AVAILABLE 2023-05-15 13:58:08.212 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:58:08.192+08:00 and status AVAILABLE 2023-05-15 13:58:08.214 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:58:08.214 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:58:08.214 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:58:08.215 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:58:08.217 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:58:08.211+08:00 and status AVAILABLE 2023-05-15 13:58:08.220 INFO 11 --- [pool-ocp-schedules-16,,248ff1811ee4] c.o.o.m.service.SeriesIdCleanService : HostIpSize=5, clusterSize=2, obproxyClusterSize=1 2023-05-15 13:58:08.221 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:58:08.226 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:58:08.226 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:58:08.227 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:58:08.227 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:58:08.231 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:58:08.249 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:58:08.250 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:58:08.251 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:58:08.252 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:58:08.252 INFO 11 --- [pool-overview-container-3,,2096c30fd984] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:08.253 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:58:08.294 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:58:08.306 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:58:08.306 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:58:08.308 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:58:08.308 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:58:08.320 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:58:08.320 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:58:08.320 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:58:08.329 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:58:08.335 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:58:08.335 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:58:08.337 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:58:08.339 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:58:08.347 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:58:08.358 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:58:08.375 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:58:08.380 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:58:08.381 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:58:08.386 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:58:08.390 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:58:08.393 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:58:08.398 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:58:08.399 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:58:08.407 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:58:08.412 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:58:08.415 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:58:08.416 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:58:08.436 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:58:08.450 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:58:08.453 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:58:08.464 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:58:08.465 INFO 11 --- [pool-Check host status30,,74003ca2cd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:58:08.468 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:58:08.469 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:58:08.472 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:58:08.477 INFO 11 --- [pool-Check host status13,,4b6653345aef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:58:08.481 INFO 11 --- [pool-Check host status8,,7f9068fd996d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:58:08.483 INFO 11 --- [pool-Check host status14,,d14a63674d5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:58:08.501 INFO 11 --- [pool-scanner--15967797-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:58:08.550 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:58:08.563 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:08.564 INFO 11 --- [pool-Check host status1,,e817f4f3fec6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:58:08.564 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:08.579 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:58:08.579 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:58:08.579 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:58:08.579 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:58:08.579 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:58:08.584 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:58:08.584 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:58:08.584 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:58:08.585 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:58:08.585 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:58:08.607 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:58:08.609 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:58:08.610 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:58:08.610 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:58:08.611 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:58:08.625 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:58:08.626 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:58:08.626 INFO 11 --- [pool-Check host status17,,9370d2434adb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:58:08.627 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:58:08.627 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:58:08.627 INFO 11 --- [pool-Check host status28,,763f141bf47f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status19,,3c29acc820c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status2,,ba1c97da7e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:58:08.630 INFO 11 --- [pool-Check host status3,,560647409e92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:58:09.178 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,1e6c48f6189241e0,46f0d5353ff8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=1e6c48f6189241e0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:09.182 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,1e6c48f6189241e0,46f0d5353ff8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=1e6c48f6189241e0, duration=5 ms] 2023-05-15 13:58:10.820 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f527c1a9d0dd41a5,7dd0a1391d30] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=f527c1a9d0dd41a5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:10.825 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f527c1a9d0dd41a5,7dd0a1391d30] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=f527c1a9d0dd41a5, duration=6 ms] 2023-05-15 13:58:11.564 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:11.565 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:11.644 INFO 11 --- [pool-ocp-schedules-19,,b0d3fbb287e6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:11.644 INFO 11 --- [pool-ocp-schedules-19,,b0d3fbb287e6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:12.238 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:14.195 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130220, nodeCount=1000, affectRows=1000 2023-05-15 13:58:14.195 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130220, nodeCount=1000, affectRows=1000 2023-05-15 13:58:14.197 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130220, nodeCount=690, affectRows=690 2023-05-15 13:58:14.201 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130220, nodeCount=1000, validCount=1000 2023-05-15 13:58:14.203 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130220, nodeCount=690, validCount=690 2023-05-15 13:58:14.204 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130220, nodeCount=1000, validCount=1000 2023-05-15 13:58:14.565 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:14.567 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:14.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:14.832957+08:00, lastProcessTime=2023-05-15T13:58:04.832608+08:00 2023-05-15 13:58:14.855 INFO 11 --- [alarm-detect-6,,7e905b8b0bdd] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130274 ~ 1684130284 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130280=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:14.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:14.832957+08:00 2023-05-15 13:58:15.832 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:15.845 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:15.856 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:15.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:16.823 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:17.174 INFO 11 --- [pool-ocp-schedules-20,,430906cccfad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:17.567 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:17.568 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:18.253 INFO 11 --- [pool-overview-container-4,,44cca78dbdfa] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:18.507 INFO 11 --- [pool-scanner--15967897-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:20.159 INFO 11 --- [pool-scanner--15967898-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:20.180 INFO 11 --- [pool-scanner--15967899-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:20.202 INFO 11 --- [pool-scanner--15967900-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:20.225 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.230 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.272 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.277 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.290 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.293 INFO 11 --- [pool-ocp-schedules-3,,79e28cf26403] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:20.568 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:20.569 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:20.696 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:21.0, nextRun=2023-05-15 13:58:20.0, currentDate=Mon May 15 13:58:20 CST 2023 2023-05-15 13:58:20.696 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:57:21.0, nextRunTime=2023-05-15 13:58:20.0} 2023-05-15 13:58:20.701 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:58:20.701 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:20.701 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:21.171 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131799, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:21.174 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131799, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:58:21.168+08:00, endTime=null} 2023-05-15 13:58:21.193 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e55e8dd9b97b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131799, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116212, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:21.168+08:00, sub_task_instance_id=131799}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:21.195 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e55e8dd9b97b] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:58:21.196 INFO 11 --- [ForkJoinPool-65984-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:58:21.196 INFO 11 --- [ForkJoinPool-65984-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:58:21.201 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e55e8dd9b97b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131799, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:21.777 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:21.780 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:21.782 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:22.109 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116212, status:SUCCESS 2023-05-15 13:58:22.110 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116212, status:SUCCESS 2023-05-15 13:58:23.362 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,ca98604e71a24ac9,cc5e1566dd26] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ca98604e71a24ac9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:23.367 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,ca98604e71a24ac9,cc5e1566dd26] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ca98604e71a24ac9, duration=6 ms] 2023-05-15 13:58:23.570 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:23.571 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:24.800 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,b2681d7fb34242eb,b2b8998915bf] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=b2681d7fb34242eb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:24.806 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,b2681d7fb34242eb,b2b8998915bf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=b2681d7fb34242eb, duration=6 ms] 2023-05-15 13:58:24.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:24.831042+08:00, lastProcessTime=2023-05-15T13:58:14.832957+08:00 2023-05-15 13:58:24.861 INFO 11 --- [alarm-detect-6,,42250cbf5d92] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130284 ~ 1684130294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.542, values={1684130290=245.542}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:24.864 INFO 11 --- [alarm-detect-3,,dcb6ae1441d7] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130234 ~ 1684130294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388677.988, values={1684130280=388677.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:58:24.865 INFO 11 --- [alarm-detect-3,,dcb6ae1441d7] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130234 ~ 1684130294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993475.358, values={1684130280=993475.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:24.867 INFO 11 --- [alarm-detect-15,,9151d642d31e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130234 ~ 1684130294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388677.938, values={1684130280=388677.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:58:24.868 INFO 11 --- [alarm-detect-15,,9151d642d31e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130234 ~ 1684130294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993475.107, values={1684130280=993475.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:24.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:24.831042+08:00 2023-05-15 13:58:25.136 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,afd286c022db4ad5,ef7906e01185] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=afd286c022db4ad5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:25.142 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,afd286c022db4ad5,ef7906e01185] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=afd286c022db4ad5, duration=7 ms] 2023-05-15 13:58:25.838 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:25.848 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:25.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:25.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:25.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:26.0, nextRun=2023-05-15 13:58:25.0, currentDate=Mon May 15 13:58:25 CST 2023 2023-05-15 13:58:25.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:57:26.0, nextRunTime=2023-05-15 13:58:25.0} 2023-05-15 13:58:25.886 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:58:25.886 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:25.886 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:26.264 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131800, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:26.266 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131800, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:58:26.261+08:00, endTime=null} 2023-05-15 13:58:26.284 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131800, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116213, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:26.261+08:00, sub_task_instance_id=131800}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:58:26.321 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:58:26.322 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:58:26.323 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:58:26.324 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:58:26.324 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:58:26.330 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5a98bdd7556c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131800, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:26.571 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:26.572 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:26.826 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:27.195 INFO 11 --- [pool-ocp-schedules-12,,c6a21ea7dbcd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:27.214 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116213, status:SUCCESS 2023-05-15 13:58:27.216 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116213, status:SUCCESS 2023-05-15 13:58:27.248 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:28.253 INFO 11 --- [pool-overview-container-1,,ce0ac698aa1c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:28.506 INFO 11 --- [pool-scanner--15968001-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:29.572 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:29.574 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:30.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:57:31.0, nextRun=2023-05-15 13:58:30.0, currentDate=Mon May 15 13:58:30 CST 2023 2023-05-15 13:58:30.030 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:57:31.0, nextRunTime=2023-05-15 13:58:30.0} 2023-05-15 13:58:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:58:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:30.037 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:30.049 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 13:58:30.0, currentDate=Mon May 15 13:58:30 CST 2023 2023-05-15 13:58:30.049 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 13:58:30.0} 2023-05-15 13:58:30.052 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:58:30.052 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:58:30.052 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:58:30.352 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131802, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:30.354 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131802, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:58:30.350+08:00, endTime=null} 2023-05-15 13:58:30.369 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131803, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:30.395 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131803, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:58:30.361+08:00, endTime=null} 2023-05-15 13:58:30.414 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131801, operation:EXECUTE, state: RUNNING 2023-05-15 13:58:30.417 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131801, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:58:30.410+08:00, endTime=null} 2023-05-15 13:58:30.421 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131802, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116214, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:30.350+08:00, sub_task_instance_id=131802}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:58:30.425 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,674159e34dd2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131803, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116215, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:30.361+08:00, sub_task_instance_id=131803}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:58:30.425 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,674159e34dd2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:58:30.447 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131801, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116214, task_operation=execute, latest_execution_start_time=2023-05-15T13:58:30.410+08:00, sub_task_instance_id=131801}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:58:30.451 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:58:30.451 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:58:30.451 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:58:30.452 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:58:30.452 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:58:30.454 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:58:30.468 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.468 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.469 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.469 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.470 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.470 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.470 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.470 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.473 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.474 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.474 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.474 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.479 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.480 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.480 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.481 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.480 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:58:30.481 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:58:30.484 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:58:30.484 INFO 11 --- [pool-3242467-thread-3,fb27d19334114ed7,bfa17c590a34] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:58:30.487 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:58:30.488 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.488 INFO 11 --- [pool-3242467-thread-5,fb27d19334114ed7,a7b18e58cd6a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:58:30.488 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.489 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.489 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.489 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.489 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.490 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.491 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.491 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:58:30.492 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:58:30.492 INFO 11 --- [pool-3242467-thread-1,fb27d19334114ed7,f0f98ab4ebb9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:58:30.493 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:58:30.493 INFO 11 --- [pool-3242467-thread-2,fb27d19334114ed7,8bb7f1dd5f80] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:58:30.494 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.494 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.494 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.494 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.495 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.495 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:58:30.495 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.495 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.495 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.496 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.496 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.496 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.496 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.496 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.496 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.497 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:58:30.499 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:58:30.499 INFO 11 --- [pool-3242467-thread-4,fb27d19334114ed7,3144b68f0916] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:58:30.500 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:58:30.500 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.500 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.500 INFO 11 --- [pool-3242467-thread-6,fb27d19334114ed7,f2a3ffe4fcd9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:58:30.500 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:58:30.500 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.501 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,674159e34dd2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:58:30.501 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.501 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.504 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.504 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.505 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,674159e34dd2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131803, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:30.505 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.510 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.510 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.510 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.520 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.520 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.521 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.524 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.524 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.525 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.534 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.534 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.535 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.535 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.535 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.535 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.545 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.545 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.546 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:58:30.548 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.548 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.549 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.562 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:58:30.562 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.562 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.563 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:58:30.572 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.572 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.572 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:58:30.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:58:30.590 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:58:30.596 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.597 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.598 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.598 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.599 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.603 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.604 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.604 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.609 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.609 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.609 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.609 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.610 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.611 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.611 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.612 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.626 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.626 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.626 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.627 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.627 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.627 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.627 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.628 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.628 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.629 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.636 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.636 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.637 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.653 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.654 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.654 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.654 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.655 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.656 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.656 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.657 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.670 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.670 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.671 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.672 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.672 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.672 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.673 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.673 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.673 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.676 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.677 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.677 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.695 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.695 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.695 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.696 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.696 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.697 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.697 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.698 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.699 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.700 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.701 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.701 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.702 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.706 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.706 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.707 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.708 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:58:30.712 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.712 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:58:30.713 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.713 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.713 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.713 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.715 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.715 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.715 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.734 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.735 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.735 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.735 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.736 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.737 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.737 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.738 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:58:30.745 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.746 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.746 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.746 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.747 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.751 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.751 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.752 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.752 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.752 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.753 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.753 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.754 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:58:30.754 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:58:30.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:58:30.776 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.776 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.777 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.782 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:30.782 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.783 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.783 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.783 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:30.786 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.786 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.787 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.794 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.797 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:58:30.798 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:30.798 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.799 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:58:30.802 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.804 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:58:30.805 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.805 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.805 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:58:30.810 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.812 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:58:30.813 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:58:30.813 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:58:30.814 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.817 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,185dfe523c98] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131802, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:30.817 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:58:30.818 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.818 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.819 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:58:30.834 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.836 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:58:30.837 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:30.837 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:30.837 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:58:30.845 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:58:30.846 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:58:30.846 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:58:30.846 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:58:30.851 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,645d16258b78] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131801, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:58:31.375 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116214, status:SUCCESS 2023-05-15 13:58:31.376 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116214, status:SUCCESS 2023-05-15 13:58:31.385 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116215, status:SUCCESS 2023-05-15 13:58:31.386 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116215, status:SUCCESS 2023-05-15 13:58:31.649 INFO 11 --- [pool-ocp-schedules-15,,da85665d15be] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:31.649 INFO 11 --- [pool-ocp-schedules-15,,da85665d15be] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:31.960 INFO 11 --- [pool-ocp-schedules-8,,c6e10d5f741f] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:58:31.963 INFO 11 --- [pool-ocp-schedules-8,,c6e10d5f741f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:58:31.977 INFO 11 --- [pool-ocp-schedules-14,,4ca6f099915f] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:58:31.979 INFO 11 --- [pool-ocp-schedules-14,,4ca6f099915f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:58:31.986 INFO 11 --- [pool-ocp-schedules-6,,30b34c536d5e] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:58:31.988 INFO 11 --- [pool-ocp-schedules-6,,30b34c536d5e] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:58:31.990 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:31.991 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:31.991 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:31.991 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:31.992 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:31.992 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:31.992 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:31.992 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:31.992 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:31.993 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:31.993 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:31.994 INFO 11 --- [ForkJoinPool-65985-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:58:31.998 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:31.998 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:31.999 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.002 INFO 11 --- [ForkJoinPool-65985-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:58:32.021 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.022 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.022 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.022 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.023 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.023 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.024 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:58:32.054 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.054 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.054 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.055 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.059 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.060 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.060 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:58:32.080 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:32.081 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.081 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.081 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.081 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.082 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.082 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.083 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:58:32.084 WARN 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:58:32.084 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.084 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.084 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:58:32.085 INFO 11 --- [ForkJoinPool-65987-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:58:32.086 INFO 11 --- [ForkJoinPool-65987-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:58:32.086 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:58:32.094 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.095 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:58:32.107 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.120 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:32.120 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.120 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.120 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.121 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.122 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:58:32.122 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.122 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.132 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:58:32.132 INFO 11 --- [ForkJoinPool-65987-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:58:32.137 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:32.138 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.139 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.139 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.139 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.144 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.144 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.144 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:58:32.148 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.148 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.149 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:58:32.151 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:58:32.169 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.169 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:58:32.189 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.189 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:58:32.204 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.204 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:58:32.219 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.220 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:58:32.233 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.233 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:58:32.251 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:58:32.269 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:58:32.270 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.271 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.271 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.272 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.276 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:58:32.276 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:58:32.277 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:58:32.296 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:58:32.297 INFO 11 --- [ForkJoinPool-65987-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:58:32.574 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:32.576 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:34.836 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:34.835871+08:00, lastProcessTime=2023-05-15T13:58:24.831042+08:00 2023-05-15 13:58:34.925 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:34.835871+08:00 2023-05-15 13:58:34.930 INFO 11 --- [alarm-detect-7,,c1045c386546] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130294 ~ 1684130304 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130300=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:35.576 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:35.577 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:35.843 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:35.850 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:35.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:35.863 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:36.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:37.216 INFO 11 --- [pool-ocp-schedules-13,,2d2ebb84258b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:38.254 INFO 11 --- [pool-overview-container-6,,14d5eda85786] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:38.507 INFO 11 --- [pool-scanner--15968109-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:38.578 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:38.579 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:39.105 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,c0db1fb3dd454ccd,bb0b9b97e049] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=c0db1fb3dd454ccd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:58:39.107 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,c0db1fb3dd454ccd,bb0b9b97e049] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=c0db1fb3dd454ccd, duration=3 ms] 2023-05-15 13:58:41.579 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:41.580 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:42.250 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:44.580 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:44.581 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:44.713 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,07642fd9cbb64aab,1c13d6b308d1] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=07642fd9cbb64aab, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:44.718 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,07642fd9cbb64aab,1c13d6b308d1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=07642fd9cbb64aab, duration=5 ms] 2023-05-15 13:58:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:44.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:44.830749+08:00, lastProcessTime=2023-05-15T13:58:34.835871+08:00 2023-05-15 13:58:44.867 INFO 11 --- [alarm-detect-12,,a01079bc4c02] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130304 ~ 1684130314 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130310=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:44.901 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:44.830749+08:00 2023-05-15 13:58:45.460 INFO 11 --- [pool-ocp-schedules-4,,f90d31ddf6b5] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:58:45.847 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:45.852 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:45.863 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:45.865 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:46.692 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689} 2023-05-15 13:58:46.724 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.725 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.726 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.726 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.731 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.731 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.732 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.732 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.733 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.734 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.735 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:46.628, end=2023-05-15T13:56:46.689},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 13:58:46.735 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 13:58:46.735 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 13:58:46.833 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:47.243 INFO 11 --- [pool-ocp-schedules-15,,79fc3b19460c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:47.581 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:47.582 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:48.253 INFO 11 --- [pool-overview-container-3,,10383cf6cabd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:48.503 INFO 11 --- [pool-scanner--15968209-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:50.583 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:50.584 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:51.654 INFO 11 --- [pool-ocp-schedules-1,,7b41f8ba96e0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:51.654 INFO 11 --- [pool-ocp-schedules-1,,7b41f8ba96e0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:58:51.779 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:51.783 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:51.785 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:52.637 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,ca29a5a1a724405e,5f20f0a79ec6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ca29a5a1a724405e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:52.642 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,ca29a5a1a724405e,5f20f0a79ec6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ca29a5a1a724405e, duration=6 ms] 2023-05-15 13:58:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,24c4f01faab3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40112} 2023-05-15 13:58:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,24c4f01faab3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:58:53.584 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:53.585 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:58:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:58:54.833056+08:00, lastProcessTime=2023-05-15T13:58:44.830749+08:00 2023-05-15 13:58:54.853 INFO 11 --- [alarm-detect-7,,233546ef255e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130314 ~ 1684130324 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130320=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:58:54.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:58:54.833056+08:00 2023-05-15 13:58:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,535cfbcecf63] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:58:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,5c6329fc5052] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:58:55.849 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:55.855 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:55.865 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:55.867 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:56.585 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:56.587 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:58:56.836 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:57.253 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:58:57.265 INFO 11 --- [pool-ocp-schedules-14,,0568003a2ec3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:58:58.254 INFO 11 --- [pool-overview-container-4,,aa9f28156933] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:58:58.504 INFO 11 --- [pool-scanner--15968303-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:58:59.034 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,09a7775190a04a36,2abcd4178a17] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=09a7775190a04a36, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:58:59.039 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,09a7775190a04a36,2abcd4178a17] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=09a7775190a04a36, duration=5 ms] 2023-05-15 13:58:59.587 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:58:59.588 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:59:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 13:59:00.001 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 13:59:00.001 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130340 2023-05-15 13:59:00.051 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 13:59:00.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 13:59:00.0, currentDate=Mon May 15 13:59:00 CST 2023 2023-05-15 13:59:00.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 13:59:00.0} 2023-05-15 13:59:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 13:59:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:00.922 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 13:59:00.0, currentDate=Mon May 15 13:59:00 CST 2023 2023-05-15 13:59:00.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 13:59:00.0} 2023-05-15 13:59:00.936 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:59:00.936 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:00.936 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:00.943 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 13:59:00.0, currentDate=Mon May 15 13:59:00 CST 2023 2023-05-15 13:59:00.943 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 13:59:00.0} 2023-05-15 13:59:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 13:59:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:00.957 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:00.963 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:30.0, nextRun=2023-05-15 13:59:00.0, currentDate=Mon May 15 13:59:00 CST 2023 2023-05-15 13:59:00.963 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:58:30.0, nextRunTime=2023-05-15 13:59:00.0} 2023-05-15 13:59:00.966 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:59:00.966 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:00.966 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:00.972 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 13:59:00.0, currentDate=Mon May 15 13:59:00 CST 2023 2023-05-15 13:59:00.972 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 13:59:00.0} 2023-05-15 13:59:00.975 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 13:59:00.975 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:00.975 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,34c4cd598b8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130400, interval=60 2023-05-15 13:59:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,4646d038ada7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130400, interval=60 2023-05-15 13:59:01.526 INFO 11 --- [pool-prometheus-minute-schedule-1,,4646d038ada7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 13:59:01.526 INFO 11 --- [pool-prometheus-minute-schedule-1,,4646d038ada7] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130341, count=6, elapsed=1682446211185 2023-05-15 13:59:01.527 INFO 11 --- [pool-prometheus-minute-schedule-2,,34c4cd598b8c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 13:59:01.527 INFO 11 --- [pool-prometheus-minute-schedule-2,,34c4cd598b8c] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130341, count=8, elapsed=1682446211186 2023-05-15 13:59:01.868 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131808, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:01.870 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131808, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T13:59:01.866+08:00, endTime=null} 2023-05-15 13:59:01.885 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131806, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:01.887 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131806, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T13:59:01.882+08:00, endTime=null} 2023-05-15 13:59:01.895 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bf00ccfaf8ae] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131808, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116220, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:01.866+08:00, sub_task_instance_id=131808}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:01.903 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131807, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:01.906 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131807, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:59:01.899+08:00, endTime=null} 2023-05-15 13:59:01.912 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bf00ccfaf8ae] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131808, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:01.912 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131806, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116218, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:01.882+08:00, sub_task_instance_id=131806}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:01.918 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131804, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:01.919 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131804, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T13:59:01.915+08:00, endTime=null} 2023-05-15 13:59:01.920 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116220, status:SUCCESS 2023-05-15 13:59:01.921 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116220, status:SUCCESS 2023-05-15 13:59:01.928 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,30ffd26ed108] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131807, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116219, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:01.899+08:00, sub_task_instance_id=131807}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:01.928 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,30ffd26ed108] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:59:01.932 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131805, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:01.934 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131805, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T13:59:01.929+08:00, endTime=null} 2023-05-15 13:59:01.935 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 13:59:01.935 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 13:59:01.936 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 13:59:01.936 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 13:59:01.939 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,614a1b17aaf9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131804, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116216, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:01.915+08:00, sub_task_instance_id=131804}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:01.941 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,80caa928c790] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131806, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:01.947 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 13:59:01.947 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 13:59:01.947 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 13:59:01.947 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 13:59:01.947 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:59:01.947 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 13:59:01.947 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:59:01.947 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:59:01.948 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:59:01.949 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:59:01.949 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:59:01.953 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:59:01.953 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,614a1b17aaf9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131804, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:01.953 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,77545571e66c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131805, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116217, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:01.929+08:00, sub_task_instance_id=131805}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:01.954 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:59:01.954 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:59:01.954 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:59:01.954 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:59:01.961 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.962 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.962 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.962 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.962 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.963 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.963 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.963 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.963 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.964 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.965 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.965 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.966 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:59:01.967 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:59:01.968 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.968 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.968 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-2,fb27d19334114ed7,f8999d68725f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-1,fb27d19334114ed7,98a29e44e71d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-3,fb27d19334114ed7,5e68649df697] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:59:01.969 INFO 11 --- [pool-3242634-thread-4,fb27d19334114ed7,5a96db2d7ee5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:59:01.970 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.970 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.970 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.971 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:01.974 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,77545571e66c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 13:59:01.974 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,77545571e66c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 13:59:01.975 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.975 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.975 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:59:01.976 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:01.976 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:01.977 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:59:01.979 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 13:59:01.979 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 13:59:01.979 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 13:59:01.979 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,77545571e66c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131805, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:01.983 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 13:59:01.983 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:59:01.983 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:59:01.983 INFO 11 --- [pool-3242634-thread-5,fb27d19334114ed7,b0b523c7b08f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:59:01.984 INFO 11 --- [pool-3242634-thread-6,fb27d19334114ed7,21ca58e6a1c7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:59:01.984 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 13:59:01.984 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,30ffd26ed108] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:59:01.988 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 13:59:01.989 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 1 2023-05-15 13:59:01.991 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,30ffd26ed108] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131807, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:01.994 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=494 2023-05-15 13:59:01.996 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=495 2023-05-15 13:59:01.997 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 13:59:02.000 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=495 2023-05-15 13:59:02.012 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm rootservice_log_alarm null 2023-05-15 13:59:02.588 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:02.589 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:02.949 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116216, status:SUCCESS 2023-05-15 13:59:02.950 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116216, status:SUCCESS 2023-05-15 13:59:02.961 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116217, status:SUCCESS 2023-05-15 13:59:02.962 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116217, status:SUCCESS 2023-05-15 13:59:02.972 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116218, status:SUCCESS 2023-05-15 13:59:02.973 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116218, status:SUCCESS 2023-05-15 13:59:02.985 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116219, status:SUCCESS 2023-05-15 13:59:02.986 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116219, status:SUCCESS 2023-05-15 13:59:04.723 INFO 11 --- [pool-ocp-schedules-2,,8e0e02bf99e5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 13:59:04.734 INFO 11 --- [pool-ocp-schedules-2,,8e0e02bf99e5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 13:59:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:04.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:04.829359+08:00, lastProcessTime=2023-05-15T13:58:54.833056+08:00 2023-05-15 13:59:04.881 INFO 11 --- [alarm-detect-3,,e2f6fd2c1f61] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130324 ~ 1684130334 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130330=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:04.901 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:04.829359+08:00 2023-05-15 13:59:05.080 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:05.0, nextRun=2023-05-15 13:59:05.0, currentDate=Mon May 15 13:59:05 CST 2023 2023-05-15 13:59:05.080 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:58:05.0, nextRunTime=2023-05-15 13:59:05.0} 2023-05-15 13:59:05.084 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 13:59:05.084 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:05.084 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:05.589 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:05.590 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:05.855 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:05.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:05.867 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:05.869 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:05.897 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,9eed090a141f4436,11a653fa4086] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=9eed090a141f4436, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:05.903 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,9eed090a141f4436,11a653fa4086] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=9eed090a141f4436, duration=6 ms] 2023-05-15 13:59:06.005 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131809, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:06.007 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131809, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T13:59:06.001+08:00, endTime=null} 2023-05-15 13:59:06.028 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9b034f4e0cd9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131809, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116221, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:06.001+08:00, sub_task_instance_id=131809}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:06.053 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,f1e5a3b86c3e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:59:06.053 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,49779c4f09ae] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:59:06.053 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,1d81dd2b46c2] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 13:59:06.091 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,49779c4f09ae] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 13:59:06.092 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,1d81dd2b46c2] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 13:59:06.098 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,f1e5a3b86c3e] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 13:59:06.110 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9b034f4e0cd9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131809, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:06.838 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:07.112 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116221, status:SUCCESS 2023-05-15 13:59:07.113 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116221, status:SUCCESS 2023-05-15 13:59:07.285 INFO 11 --- [pool-ocp-schedules-19,,96fc44acd789] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:08.255 INFO 11 --- [pool-overview-container-1,,9e5aa1ab3eca] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 13:59:08.255 INFO 11 --- [pool-overview-container-1,,9e5aa1ab3eca] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=494 2023-05-15 13:59:08.505 INFO 11 --- [pool-scanner--15968396-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 13:59:08.590 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:08.591 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:08.651 INFO 11 --- [pool-ocp-schedules-11,,0be8de8103c0] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 13:59:08.651 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:59:08.651 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:59:08.651 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:59:08.651 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:59:08.652 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:59:08.658 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:59:08.658 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:59:08.658 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:59:08.658 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:59:08.658 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:59:08.686 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 13:59:08.688 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 13:59:08.688 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T13:59:08.688438841+08:00 2023-05-15 13:59:08.689 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:59:08.689+08:00 2023-05-15 13:59:08.689 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 13:59:08.689 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 13:59:08.691 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T13:59:08.690797130+08:00 2023-05-15 13:59:08.691 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:59:08.691+08:00 2023-05-15 13:59:08.691 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T13:59:08.691472319+08:00 2023-05-15 13:59:08.691 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 13:59:08.691 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:59:08.691+08:00 2023-05-15 13:59:08.692 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T13:59:08.691684824+08:00 2023-05-15 13:59:08.692 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:59:08.692+08:00 2023-05-15 13:59:08.692 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:59:08.689+08:00 and status AVAILABLE 2023-05-15 13:59:08.694 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T13:59:08.693671565+08:00 2023-05-15 13:59:08.694 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:59:08.694+08:00 2023-05-15 13:59:08.695 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:59:08.691+08:00 and status AVAILABLE 2023-05-15 13:59:08.695 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:59:08.691+08:00 and status AVAILABLE 2023-05-15 13:59:08.696 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:59:08.692+08:00 and status AVAILABLE 2023-05-15 13:59:08.698 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:59:08.694+08:00 and status AVAILABLE 2023-05-15 13:59:08.734 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 13:59:08.735 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 13:59:08.735 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 13:59:08.735 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 13:59:08.735 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 13:59:08.757 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130348 0 0 , error= 2023-05-15 13:59:08.761 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130348 0 0 , error= 2023-05-15 13:59:08.761 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130348 0 0 , error= 2023-05-15 13:59:08.761 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130348 0 0 , error= 2023-05-15 13:59:08.766 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T13:59:08.765+08:00 2023-05-15 13:59:08.766 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T13:59:08.765+08:00 2023-05-15 13:59:08.766 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T13:59:08.765+08:00 2023-05-15 13:59:08.766 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T13:59:08.766+08:00 2023-05-15 13:59:08.771 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T13:59:08.765+08:00 and status AVAILABLE 2023-05-15 13:59:08.771 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T13:59:08.766+08:00 and status AVAILABLE 2023-05-15 13:59:08.771 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T13:59:08.765+08:00 and status AVAILABLE 2023-05-15 13:59:08.771 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T13:59:08.765+08:00 and status AVAILABLE 2023-05-15 13:59:08.795 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 13:59:08.795 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 13:59:08.795 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 13:59:08.795 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 13:59:08.806 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:59:08.806 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:59:08.806 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:59:08.807 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:59:08.834 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:59:08.835 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:59:08.836 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:59:08.839 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:59:08.878 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 13:59:08.889 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 13:59:08.893 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 13:59:08.893 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 13:59:08.903 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 13:59:08.907 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 13:59:08.916 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:59:08.922 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 13:59:08.923 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:59:08.927 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:59:08.935 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 13:59:08.956 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:59:08.966 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 13:59:08.967 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 13:59:08.979 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 13:59:08.980 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 13:59:08.982 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 13:59:08.994 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 13:59:09.016 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:59:09.017 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:59:09.017 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:59:09.034 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 13:59:09.048 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 13:59:09.058 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 13:59:09.065 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:59:09.071 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 13:59:09.072 INFO 11 --- [pool-Check host status12,,db31bb7574ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 13:59:09.077 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 13:59:09.084 INFO 11 --- [pool-Check host status10,,2466b4fdc035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 13:59:09.093 INFO 11 --- [pool-Check host status22,,6297c5b1fcfa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 13:59:09.143 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 13:59:09.158 INFO 11 --- [pool-Check host status9,,6123e072d7a0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 13:59:09.701 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130349 0 0 , error= 2023-05-15 13:59:09.701 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T13:59:09.701+08:00 2023-05-15 13:59:09.707 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T13:59:09.701+08:00 and status AVAILABLE 2023-05-15 13:59:09.710 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 13:59:09.722 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:59:09.753 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 13:59:09.814 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 13:59:09.829 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 13:59:09.854 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 13:59:09.904 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 13:59:09.915 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 13:59:09.937 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 13:59:09.943 INFO 11 --- [pool-ocp-schedules-6,,e3f8ba1d38ac] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40112} 2023-05-15 13:59:09.944 INFO 11 --- [pool-ocp-schedules-6,,e3f8ba1d38ac] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:59:09.992 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 13:59:10.009 INFO 11 --- [pool-Check host status26,,5fbe9c3c066b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 13:59:10.016 INFO 11 --- [pool-ocp-schedules-12,,d2272bc2856d] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 13:59:10.033 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 13:59:10.033 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 13:59:10.033 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 13:59:10.033 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 13:59:10.033 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 13:59:10.039 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 13:59:10.039 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 13:59:10.039 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 13:59:10.039 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 13:59:10.040 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 13:59:10.067 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 13:59:10.067 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 13:59:10.070 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 13:59:10.071 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 13:59:10.072 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status15,,cab9d7d307a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status25,,d469cd54843e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 13:59:10.088 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:59:10.089 INFO 11 --- [pool-Check host status18,,62310e336675] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 13:59:10.092 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 13:59:10.092 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:59:10.092 INFO 11 --- [pool-Check host status16,,1b108acf6d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 13:59:10.109 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 13:59:10.109 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:59:10.109 INFO 11 --- [pool-Check host status24,,e4a211415d4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 13:59:10.733 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,2826499e38264449,9ae78b08de4c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=2826499e38264449, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:10.737 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,2826499e38264449,9ae78b08de4c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=2826499e38264449, duration=4 ms] 2023-05-15 13:59:11.592 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:11.594 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:11.697 INFO 11 --- [pool-ocp-schedules-15,,c9790485a416] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:11.697 INFO 11 --- [pool-ocp-schedules-15,,c9790485a416] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:12.255 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:13.139 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f4e41b96181f42d8,fd8727c730e0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=f4e41b96181f42d8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:59:13.141 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f4e41b96181f42d8,fd8727c730e0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=f4e41b96181f42d8, duration=2 ms] 2023-05-15 13:59:13.914 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8698d533d3484edf,215baa569916] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/loginKey?null, client=192.168.80.229, traceId=8698d533d3484edf, method=SuccessResponse com.oceanbase.ocp.server.controller.InfoController.getLoginKey(), args=] 2023-05-15 13:59:13.915 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8698d533d3484edf,215baa569916] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/loginKey client=192.168.80.229, traceId=8698d533d3484edf, duration=1 ms] 2023-05-15 13:59:14.102 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,,] c.s.h.CustomAuthenticationSuccessHandler : Authentication successful for /api/v2/iam/login 2023-05-15 13:59:14.118 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,,] c.s.h.CustomAuthenticationSuccessHandler : Redirecting to saved url: / 2023-05-15 13:59:14.221 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,4046376c2e3047f2,8a375d5429c1] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/loginKey?null, client=192.168.80.229, traceId=4046376c2e3047f2, method=SuccessResponse com.oceanbase.ocp.server.controller.InfoController.getLoginKey(), args=] 2023-05-15 13:59:14.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,4046376c2e3047f2,8a375d5429c1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/loginKey client=192.168.80.229, traceId=4046376c2e3047f2, duration=1 ms] 2023-05-15 13:59:14.223 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ee94635d571542f0,bf4f4d3c1ff4] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=2&tenantId=6&status=RUNNING, client=192.168.80.229, traceId=ee94635d571542f0, method=PaginatedResponse com.oceanbase.ocp.server.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long), args=Page request [number: 0, size 2000, sort: id: DESC],RUNNING,MANUAL,2,6,] 2023-05-15 13:59:14.223 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,7fbd9b414cca4c92,fa614a38d118] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=192.168.80.229, traceId=7fbd9b414cca4c92, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List), args=Page request [number: 0, size 2000, sort: id: DESC],] 2023-05-15 13:59:14.227 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ee94635d571542f0,bf4f4d3c1ff4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=192.168.80.229, traceId=ee94635d571542f0, duration=7 ms] 2023-05-15 13:59:14.228 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,b556e289251646e9,bd9bfb3e9b05] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=192.168.80.229, traceId=b556e289251646e9, method=SuccessResponse com.oceanbase.ocp.server.controller.ProfileController.userInfo(), args=] 2023-05-15 13:59:14.231 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,9ae10592a2ee4a3b,c30046e65da9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/maz/state?null, client=192.168.80.229, traceId=9ae10592a2ee4a3b, method=SuccessResponse com.oceanbase.ocp.server.controller.MultiAzoneController.state(), args=] 2023-05-15 13:59:14.232 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,9ae10592a2ee4a3b,c30046e65da9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/maz/state client=192.168.80.229, traceId=9ae10592a2ee4a3b, duration=2 ms] 2023-05-15 13:59:14.232 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,b556e289251646e9,bd9bfb3e9b05] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=192.168.80.229, traceId=b556e289251646e9, duration=4 ms] 2023-05-15 13:59:14.233 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130280, nodeCount=690, affectRows=690 2023-05-15 13:59:14.233 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6?id=2&tenantId=6, client=192.168.80.229, traceId=457fffdb8ac2455d, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getTenant(Long,Long), args=2,6,] 2023-05-15 13:59:14.234 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,2dbcc2e9fba045c2,d24ee97fb182] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=192.168.80.229, traceId=2dbcc2e9fba045c2, method=SuccessResponse com.oceanbase.ocp.server.controller.PropertyController.getSystemInfo(), args=] 2023-05-15 13:59:14.234 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,2dbcc2e9fba045c2,d24ee97fb182] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=192.168.80.229, traceId=2dbcc2e9fba045c2, duration=1 ms] 2023-05-15 13:59:14.236 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130280, nodeCount=1000, affectRows=1000 2023-05-15 13:59:14.236 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130280, nodeCount=690, validCount=690 2023-05-15 13:59:14.236 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,6b70592af8b5426d,583480301aae] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/mc/info?null, client=192.168.80.229, traceId=6b70592af8b5426d, method=SuccessResponse com.oceanbase.ocp.server.controller.OcpMultiClusterInfoController.info(), args=] 2023-05-15 13:59:14.236 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,6b70592af8b5426d,583480301aae] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/mc/info client=192.168.80.229, traceId=6b70592af8b5426d, duration=0 ms] 2023-05-15 13:59:14.236 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,96050fb6cdad4962,facba1fa7ae9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=192.168.80.229, traceId=96050fb6cdad4962, method=Map com.oceanbase.ocp.server.controller.InfoController.info(), args=] 2023-05-15 13:59:14.236 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,96050fb6cdad4962,facba1fa7ae9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=192.168.80.229, traceId=96050fb6cdad4962, duration=0 ms] 2023-05-15 13:59:14.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,c9be912a60804ce0,146e65d4e5a8] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.80.229, traceId=c9be912a60804ce0, method=OffsetDateTime com.oceanbase.ocp.server.controller.InfoController.time(), args=] 2023-05-15 13:59:14.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,c9be912a60804ce0,146e65d4e5a8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.80.229, traceId=c9be912a60804ce0, duration=0 ms] 2023-05-15 13:59:14.241 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130280, nodeCount=1000, affectRows=1000 2023-05-15 13:59:14.242 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130280, nodeCount=1000, validCount=1000 2023-05-15 13:59:14.244 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sessions?page=1&size=10&activeOnly=false, client=192.168.80.229, traceId=444b6da9cb694574, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.listTenantSessions(Pageable,Long,Long,String,String,String,Boolean), args=Page request [number: 0, size 10, sort: id: DESC],2,6,false,] 2023-05-15 13:59:14.246 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130280, nodeCount=1000, validCount=1000 2023-05-15 13:59:14.251 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,7fbd9b414cca4c92,fa614a38d118] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=192.168.80.229, traceId=7fbd9b414cca4c92, duration=31 ms] 2023-05-15 13:59:14.255 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.255 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.256 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.256 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:14.259 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 13:59:14.259 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 13:59:14.264 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:14.265 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.265 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.265 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.266 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:14.267 ERROR 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66246) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=66246) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 13:59:14.268 ERROR 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] com.alibaba.druid.pool.DruidDataSource : {dataSource-90} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66246) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=66246) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 13:59:14.269 INFO 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] com.alibaba.druid.pool.DruidDataSource : {dataSource-90,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 13:59:14.269 INFO 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] com.alibaba.druid.pool.DruidDataSource : {dataSource-90} closing ... 2023-05-15 13:59:14.269 INFO 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] com.alibaba.druid.pool.DruidDataSource : {dataSource-90} closed 2023-05-15 13:59:14.272 ERROR 11 --- [obsdk-init-thread-7,444b6da9cb694574,fa20d028bd9b] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=66246) Get Location Cache Fail 2023-05-15 13:59:14.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.272 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66246) Get Location Cache Fail 2023-05-15 13:59:14.273 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66246) Get Location Cache Fail 2023-05-15 13:59:14.273 WARN 11 --- [http-nio-0.0.0.0-8080-exec-65,444b6da9cb694574,a2604e2d5cd3] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/sessions client=192.168.80.229, traceId=444b6da9cb694574, duration=31 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 13:59:14.273 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.MAX_IOPS, t2.MIN_IOPS FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2023-05-15 13:59:14.280 ERROR 11 --- [Druid-ConnectionPool-Create-1397426893,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66249) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=66249) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 13:59:14.284 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:14.285 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.285 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.285 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.286 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:14.300 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.300 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.301 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2023-05-15 13:59:14.319 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.319 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.320 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2023-05-15 13:59:14.323 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.323 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.323 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:14.343 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.344 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.344 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:14.368 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:14.369 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.370 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.370 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.370 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:14.385 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:14.386 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:14.386 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 13:59:14.424 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 13:59:14.424 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 13:59:14.435 ERROR 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 13:59:14.437 ERROR 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] com.alibaba.druid.pool.DruidDataSource : {dataSource-91} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 13:59:14.437 INFO 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] com.alibaba.druid.pool.DruidDataSource : {dataSource-91,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 13:59:14.437 INFO 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] com.alibaba.druid.pool.DruidDataSource : {dataSource-91} closing ... 2023-05-15 13:59:14.437 INFO 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] com.alibaba.druid.pool.DruidDataSource : {dataSource-91} closed 2023-05-15 13:59:14.440 ERROR 11 --- [obsdk-init-thread-8,457fffdb8ac2455d,783b339ad214] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail 2023-05-15 13:59:14.441 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail 2023-05-15 13:59:14.441 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail 2023-05-15 13:59:14.448 ERROR 11 --- [Druid-ConnectionPool-Create-1742539780,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66255) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=66255) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 13:59:14.458 WARN 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.o.i.tenant.TenantServiceImpl : Can not get tenant extra info from ob. exception: com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:144) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:131) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:78) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.parameter.TenantParameterServiceImpl.listTenantParameter(TenantParameterServiceImpl.java:116) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.parameter.TenantParameterServiceImpl.getTenantParameter(TenantParameterServiceImpl.java:232) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.tenant.TenantServiceImpl.attachPerfParameters(TenantServiceImpl.java:410) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.tenant.TenantServiceImpl.getTenantInfo(TenantServiceImpl.java:362) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.server.controller.ObTenantController.getTenant(ObTenantController.java:172) [classes!/:4.0.3-20230301] at com.oceanbase.ocp.server.controller.ObTenantController$$FastClassBySpringCGLIB$$8f5d377e.invoke() [classes!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) [spring-context-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) [spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.server.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:119) [classes!/:4.0.3-20230301] at sun.reflect.GeneratedMethodAccessor543.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) [spring-security-core-5.7.5.jar!/:5.7.5] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) [spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.server.controller.ObTenantController$$EnhancerBySpringCGLIB$$d343b503.getTenant() [classes!/:4.0.3-20230301] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.23.jar!/:5.3.23] at javax.servlet.http.HttpServlet.service(HttpServlet.java:670) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.23.jar!/:5.3.23] at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:111) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at com.oceanbase.ocp.core.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:106) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) [spring-session-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) [spring-session-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.6.13.jar!/:2.6.13] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at com.oceanbase.ocp.core.security.accesskey.ContentCacheRequestFilter.doFilter(ContentCacheRequestFilter.java:31) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.68.jar!/:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_312] Caused by: com.oceanbase.ocp.obsdk.exception.ConnectorInitFailedException: [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail at com.oceanbase.ocp.obsdk.connector.ObConnectors.newObConnector(ObConnectors.java:257) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.createObConnector(ObConnectors.java:236) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.getObConnector(ObConnectors.java:102) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.getDataSource(ConnectTemplate.java:329) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.initJdbcTemplate(ConnectTemplate.java:317) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.(ConnectTemplate.java:53) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectTemplate.(ObConnectTemplate.java:30) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.accessor.ObAccessors.newObAccessor(ObAccessors.java:56) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:140) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] ... 153 common frames omitted Caused by: java.util.concurrent.ExecutionException: java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_312] at com.oceanbase.ocp.obsdk.connector.ObConnectors.newObConnector(ObConnectors.java:249) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] ... 161 common frames omitted Caused by: java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] ... 1 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=66252) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 13:59:14.464 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,457fffdb8ac2455d,187ee1c6ae4c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6 client=192.168.80.229, traceId=457fffdb8ac2455d, duration=232 ms] 2023-05-15 13:59:14.594 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:14.595 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:14.889 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:14.888171+08:00, lastProcessTime=2023-05-15T13:59:04.829359+08:00 2023-05-15 13:59:14.934 INFO 11 --- [alarm-detect-12,,5f35bce89a04] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130334 ~ 1684130344 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130340=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:14.963 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:14.888171+08:00 2023-05-15 13:59:15.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:15.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:15.869 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:15.871 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:16.842 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:17.307 INFO 11 --- [pool-ocp-schedules-6,,aee989d3c936] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 13:59:17.595 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:17.596 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:17.630 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.630 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:17.632 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:17.632 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.632 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:59:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:17.667 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:17.667 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.667 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:59:17.670 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 13:59:17.670 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 13:59:17.670 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 13:59:17.670 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 13:59:17.675 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:59:17.675 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.676 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:59:17.676 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.676 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 13:59:17.676 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:17.677 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:59:17.677 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:59:17.677 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 13:59:17.701 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T13:57:02, start=2023-05-15T13:53:17, end=2023-05-15T13:55:47, updates=102, granule=120000ms, delay=225000ms 2023-05-15 13:59:17.703 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:57:02, target=2023-05-15T13:55:47, granule=120000ms, delay=75000ms) 2023-05-15 13:59:17.703 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:55:47, target=2023-05-15T13:53:17, granule=600000ms, delay=150000ms) 2023-05-15 13:59:17.703 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T13:56:48, start=2023-05-15T13:53:17, end=2023-05-15T13:55:47, updates=124, granule=120000ms, delay=211000ms 2023-05-15 13:59:17.704 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T13:56:48, start=2023-05-15T13:53:17, end=2023-05-15T13:55:47, updates=101, granule=120000ms, delay=211000ms 2023-05-15 13:59:17.705 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:56:48, target=2023-05-15T13:55:47, granule=120000ms, delay=61000ms) 2023-05-15 13:59:17.705 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:55:47, target=2023-05-15T13:51:17, granule=600000ms, delay=270000ms) 2023-05-15 13:59:17.705 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:56:48, target=2023-05-15T13:55:47, granule=120000ms, delay=61000ms) 2023-05-15 13:59:17.705 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:55:47, target=2023-05-15T13:51:17, granule=600000ms, delay=270000ms) 2023-05-15 13:59:17.706 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=119ms, period=120000ms 2023-05-15 13:59:18.256 INFO 11 --- [pool-overview-container-5,,7678442dce2c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:18.503 INFO 11 --- [pool-scanner--15968496-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:18.610 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,51652edbc4714029,ca068d9af1cb] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=51652edbc4714029, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:18.615 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,51652edbc4714029,ca068d9af1cb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=51652edbc4714029, duration=6 ms] 2023-05-15 13:59:20.329 INFO 11 --- [pool-scanner--15968497-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:20.349 INFO 11 --- [pool-scanner--15968498-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:20.367 INFO 11 --- [pool-scanner--15968499-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:20.389 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.394 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.431 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.436 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.456 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.460 INFO 11 --- [pool-ocp-schedules-10,,e3994018150a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:20.564 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:21.0, nextRun=2023-05-15 13:59:20.0, currentDate=Mon May 15 13:59:20 CST 2023 2023-05-15 13:59:20.564 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:58:21.0, nextRunTime=2023-05-15 13:59:20.0} 2023-05-15 13:59:20.567 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 13:59:20.567 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:20.567 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:20.596 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:20.597 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:21.263 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131810, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:21.266 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131810, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T13:59:21.261+08:00, endTime=null} 2023-05-15 13:59:21.294 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,c1cd661c6f72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131810, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116222, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:21.261+08:00, sub_task_instance_id=131810}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:21.297 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,c1cd661c6f72] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:59:21.297 INFO 11 --- [ForkJoinPool-65990-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:59:21.297 INFO 11 --- [ForkJoinPool-65990-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 13:59:21.303 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,c1cd661c6f72] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131810, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:21.379 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116222, status:SUCCESS 2023-05-15 13:59:21.380 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116222, status:SUCCESS 2023-05-15 13:59:21.522 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,314ed9886de14d08,f1d858be6d64] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=314ed9886de14d08, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:21.528 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,314ed9886de14d08,f1d858be6d64] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=314ed9886de14d08, duration=6 ms] 2023-05-15 13:59:21.783 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:21.786 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:21.788 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:23.598 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:23.599 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:24.830865+08:00, lastProcessTime=2023-05-15T13:59:14.888171+08:00 2023-05-15 13:59:24.857 INFO 11 --- [alarm-detect-12,,14b800de11b2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130344 ~ 1684130354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130350=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:24.862 INFO 11 --- [alarm-detect-9,,6c5cb475d0b0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130294 ~ 1684130354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388737.988, values={1684130340=388737.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:59:24.866 INFO 11 --- [alarm-detect-9,,6c5cb475d0b0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130294 ~ 1684130354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993535.358, values={1684130340=993535.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:24.863 INFO 11 --- [alarm-detect-1,,9d6542881dcb] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130294 ~ 1684130354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388737.938, values={1684130340=388737.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 13:59:24.867 INFO 11 --- [alarm-detect-1,,9d6542881dcb] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130294 ~ 1684130354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993535.107, values={1684130340=993535.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:24.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:24.830865+08:00 2023-05-15 13:59:25.717 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:26.0, nextRun=2023-05-15 13:59:25.0, currentDate=Mon May 15 13:59:25 CST 2023 2023-05-15 13:59:25.717 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:58:26.0, nextRunTime=2023-05-15 13:59:25.0} 2023-05-15 13:59:25.724 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 13:59:25.724 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:25.724 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:25.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:25.863 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:25.871 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:25.873 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:26.352 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131811, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:26.354 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131811, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T13:59:26.350+08:00, endTime=null} 2023-05-15 13:59:26.372 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131811, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116223, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:26.350+08:00, sub_task_instance_id=131811}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 13:59:26.414 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 13:59:26.415 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:59:26.415 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:59:26.416 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 13:59:26.417 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 13:59:26.423 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,ed640de33426] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131811, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:26.488 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116223, status:SUCCESS 2023-05-15 13:59:26.489 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116223, status:SUCCESS 2023-05-15 13:59:26.599 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:26.600 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:26.846 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:27.257 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:27.328 INFO 11 --- [pool-ocp-schedules-15,,f30acbfa3628] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:28.256 INFO 11 --- [pool-overview-container-2,,629922bfbc48] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:28.505 INFO 11 --- [pool-scanner--15968600-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:29.600 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:29.601 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:30.882 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:30.0, nextRun=2023-05-15 13:59:30.0, currentDate=Mon May 15 13:59:30 CST 2023 2023-05-15 13:59:30.882 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:58:30.0, nextRunTime=2023-05-15 13:59:30.0} 2023-05-15 13:59:30.890 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 13:59:30.890 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:30.890 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:30.890 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:30.900 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:01.0, nextRun=2023-05-15 13:59:30.0, currentDate=Mon May 15 13:59:30 CST 2023 2023-05-15 13:59:30.900 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:59:01.0, nextRunTime=2023-05-15 13:59:30.0} 2023-05-15 13:59:30.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 13:59:30.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 13:59:30.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 13:59:31.436 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131814, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:31.438 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131814, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T13:59:31.432+08:00, endTime=null} 2023-05-15 13:59:31.453 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131812, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:31.455 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131812, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T13:59:31.450+08:00, endTime=null} 2023-05-15 13:59:31.465 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9da8908d2fa2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131814, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116225, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:31.432+08:00, sub_task_instance_id=131814}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 13:59:31.465 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9da8908d2fa2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 13:59:31.469 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131813, operation:EXECUTE, state: RUNNING 2023-05-15 13:59:31.471 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131813, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T13:59:31.466+08:00, endTime=null} 2023-05-15 13:59:31.477 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131812, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116224, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:31.450+08:00, sub_task_instance_id=131812}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:59:31.482 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:59:31.494 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131813, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116224, task_operation=execute, latest_execution_start_time=2023-05-15T13:59:31.465+08:00, sub_task_instance_id=131813}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 13:59:31.495 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.495 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.495 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.496 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.496 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.497 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.498 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.499 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.500 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:59:31.500 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.500 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.501 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:59:31.501 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 13:59:31.501 INFO 11 --- [pool-3242794-thread-2,fb27d19334114ed7,578906526d1b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 13:59:31.502 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-3,fb27d19334114ed7,79cc3b4abd33] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-1,fb27d19334114ed7,ecc1c9818cfc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.503 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 13:59:31.504 INFO 11 --- [pool-3242794-thread-4,fb27d19334114ed7,4dce7be08188] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 13:59:31.504 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.505 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.506 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.506 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.507 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.509 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.509 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.509 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 13:59:31.511 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.511 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.512 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 13:59:31.514 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 13:59:31.514 INFO 11 --- [pool-3242794-thread-6,fb27d19334114ed7,187b7f27cc59] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 13:59:31.516 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 13:59:31.517 INFO 11 --- [pool-3242794-thread-5,fb27d19334114ed7,503c4c0d42f7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 13:59:31.517 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.517 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9da8908d2fa2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 13:59:31.518 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.518 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.518 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.519 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.519 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.519 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:59:31.522 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,9da8908d2fa2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131814, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:31.526 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.526 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.527 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.541 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.542 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.542 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.543 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.543 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.543 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.544 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.549 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.549 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.549 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 13:59:31.551 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.551 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.552 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.554 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.554 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.555 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.561 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.561 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.561 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.571 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.571 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.571 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.572 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.572 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:59:31.572 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.584 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.584 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.585 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:59:31.585 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.585 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.586 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.592 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116225, status:SUCCESS 2023-05-15 13:59:31.593 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116225, status:SUCCESS 2023-05-15 13:59:31.595 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.595 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.595 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:59:31.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.609 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.609 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.609 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:59:31.612 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:59:31.618 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 13:59:31.633 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.633 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.634 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.634 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.634 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.635 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.635 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.635 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.637 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.638 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.638 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.638 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.639 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.643 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.644 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.644 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.644 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.645 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.657 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.658 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.658 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.658 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.659 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.659 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.659 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.660 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.660 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.660 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.661 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.661 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.661 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.664 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.671 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.671 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.672 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.672 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.672 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.673 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.673 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.674 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.696 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.696 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.696 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.696 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.697 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.697 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.697 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.697 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.698 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.698 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.699 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.701 INFO 11 --- [pool-ocp-schedules-1,,52c1921738c9] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:31.701 INFO 11 --- [pool-ocp-schedules-1,,52c1921738c9] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:31.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.704 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.711 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.711 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.712 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.712 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.712 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.714 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.714 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.714 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.721 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.722 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.723 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.723 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.724 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.729 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.729 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 13:59:31.732 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:59:31.735 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.735 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.736 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.736 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.736 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.737 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.738 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.738 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 13:59:31.751 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.752 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.752 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.752 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.753 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.754 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.754 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.754 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 13:59:31.755 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 13:59:31.759 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 13:59:31.761 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.762 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.763 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.763 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.763 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.769 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.769 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.770 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.780 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:31.781 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.781 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.781 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.782 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.787 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.790 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 13:59:31.791 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:31.791 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.792 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 13:59:31.795 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.797 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 13:59:31.798 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.798 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.798 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 13:59:31.808 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.808 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.810 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 13:59:31.810 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 13:59:31.810 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:59:31.810 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 13:59:31.811 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.811 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.811 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 13:59:31.816 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3d9ba2aff986] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131813, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:31.821 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.822 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 13:59:31.823 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.823 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.823 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 13:59:31.832 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 13:59:31.833 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 13:59:31.833 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 13:59:31.833 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:59:31.838 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8fae2a511707] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131812, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 13:59:31.970 INFO 11 --- [pool-ocp-schedules-6,,0b6e2e9d2bc1] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 13:59:31.972 INFO 11 --- [pool-ocp-schedules-6,,0b6e2e9d2bc1] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:59:31.983 INFO 11 --- [pool-ocp-schedules-19,,130a73b3a94c] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 13:59:31.986 INFO 11 --- [pool-ocp-schedules-19,,130a73b3a94c] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 13:59:31.992 INFO 11 --- [pool-ocp-schedules-7,,49891a737aa5] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 13:59:31.995 INFO 11 --- [pool-ocp-schedules-7,,49891a737aa5] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 13:59:31.997 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.997 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.998 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:31.998 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:31.999 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:31.999 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.000 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.000 INFO 11 --- [ForkJoinPool-65991-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 13:59:32.004 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.004 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.004 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.005 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.010 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.010 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.010 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.014 INFO 11 --- [ForkJoinPool-65991-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 13:59:32.028 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.028 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.028 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.029 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.030 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.030 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.030 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:59:32.043 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.044 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.044 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.045 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.049 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.049 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.050 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:59:32.073 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:32.073 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.074 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.074 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.074 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.075 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.075 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.075 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:59:32.076 WARN 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 13:59:32.077 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.077 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.077 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:59:32.078 INFO 11 --- [ForkJoinPool-65993-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 13:59:32.078 INFO 11 --- [ForkJoinPool-65993-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 13:59:32.078 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:59:32.087 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.087 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:59:32.098 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.110 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:32.111 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.111 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.111 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.111 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.113 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 13:59:32.113 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.113 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.117 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:32.118 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.118 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.118 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.119 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.123 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 13:59:32.124 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.124 INFO 11 --- [ForkJoinPool-65993-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 13:59:32.124 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.124 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 13:59:32.127 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.127 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.127 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 13:59:32.129 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:59:32.148 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.148 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:59:32.171 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.171 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:59:32.185 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.185 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:59:32.200 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.200 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 13:59:32.214 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.214 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 13:59:32.229 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 13:59:32.246 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 13:59:32.247 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.247 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.247 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.248 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.254 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 13:59:32.254 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 13:59:32.254 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 13:59:32.272 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 13:59:32.272 INFO 11 --- [ForkJoinPool-65993-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 13:59:32.602 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:32.603 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:32.626 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116224, status:SUCCESS 2023-05-15 13:59:32.627 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116224, status:SUCCESS 2023-05-15 13:59:32.930 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,1a852a74b8be4d3f,59f662141cb0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=1a852a74b8be4d3f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:32.937 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,1a852a74b8be4d3f,59f662141cb0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=1a852a74b8be4d3f, duration=7 ms] 2023-05-15 13:59:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:34.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:34.829422+08:00, lastProcessTime=2023-05-15T13:59:24.830865+08:00 2023-05-15 13:59:34.860 INFO 11 --- [alarm-detect-6,,4b130e5cfbb1] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130354 ~ 1684130364 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=105.0, values={1684130360=105.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 13:59:34.862 INFO 11 --- [alarm-detect-1,,7fae9909de7b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130354 ~ 1684130364 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130360=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:34.863 INFO 11 --- [alarm-detect-6,,4b130e5cfbb1] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, currentState=Pending 2023-05-15 13:59:34.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:34.829422+08:00 2023-05-15 13:59:35.603 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:35.604 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:35.863 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:35.866 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:35.873 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:35.875 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:36.849 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:37.351 INFO 11 --- [pool-ocp-schedules-11,,db0743cbe928] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:38.256 INFO 11 --- [pool-overview-container-7,,390d5437c320] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:38.506 INFO 11 --- [pool-scanner--15968708-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:38.604 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:38.605 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:40.008 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,62e331a546e74b09,f0d22303f79b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=62e331a546e74b09, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:40.014 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,62e331a546e74b09,f0d22303f79b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=62e331a546e74b09, duration=6 ms] 2023-05-15 13:59:41.605 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:41.610 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:42.259 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:44.611 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:44.612 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:44.836 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:44.834941+08:00, lastProcessTime=2023-05-15T13:59:34.829422+08:00 2023-05-15 13:59:44.854 INFO 11 --- [alarm-detect-3,,08a94a9dcbc8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130354 ~ 1684130364 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=105.0, values={1684130360=105.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 13:59:44.860 INFO 11 --- [alarm-detect-14,,208d5c8b0818] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130364 ~ 1684130374 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.632, values={1684130370=245.632}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:44.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:44.834941+08:00 2023-05-15 13:59:45.461 INFO 11 --- [pool-ocp-schedules-1,,4b89ff3bf2c2] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 13:59:45.866 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:45.868 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:45.875 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:45.877 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:46.852 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:47.172 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,5e6acbe94a784095,1052de25a847] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=5e6acbe94a784095, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 13:59:47.175 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,5e6acbe94a784095,1052de25a847] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=5e6acbe94a784095, duration=3 ms] 2023-05-15 13:59:47.372 INFO 11 --- [pool-ocp-schedules-18,,a3d26ee37916] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:47.612 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:47.613 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:48.256 INFO 11 --- [pool-overview-container-8,,ea81a95bd188] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:48.501 INFO 11 --- [pool-scanner--15968808-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:50.613 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:50.615 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:51.707 INFO 11 --- [pool-ocp-schedules-10,,2f995131d32a] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:51.707 INFO 11 --- [pool-ocp-schedules-10,,2f995131d32a] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 13:59:51.786 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:51.788 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:51.790 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:52.508 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,c605c5e772404eb5,75fc295fef19] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=c605c5e772404eb5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:52.515 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,c605c5e772404eb5,75fc295fef19] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=c605c5e772404eb5, duration=8 ms] 2023-05-15 13:59:53.393 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.438 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.450 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.456 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.463 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.464 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.466 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.467 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.469 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.474 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.475 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.476 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.479 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.481 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.483 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.483 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.485 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.487 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.489 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.490 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.490 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.492 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.493 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.495 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.496 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.497 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.510 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,7a7e8d3d1435] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40112} 2023-05-15 13:59:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,7a7e8d3d1435] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 13:59:53.519 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 8 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.539 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 30 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.581 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 125 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:53:52.672, end=2023-05-15T13:58:53.39} 2023-05-15 13:59:53.608 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 13:59:53.608 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 13:59:53.616 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:53.616 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 13:59:54.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T13:59:54.831489+08:00, lastProcessTime=2023-05-15T13:59:44.834941+08:00 2023-05-15 13:59:54.851 INFO 11 --- [alarm-detect-4,,d93b70ea46b3] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130354 ~ 1684130364 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=105.0, values={1684130360=105.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 13:59:54.864 INFO 11 --- [alarm-detect-13,,ec6b17e6c9ac] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130374 ~ 1684130384 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.677, values={1684130380=245.677}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 13:59:54.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T13:59:54.831489+08:00 2023-05-15 13:59:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,c24294b77e0e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 13:59:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,17c46a2a79ed] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 13:59:55.868 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:55.870 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:55.877 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:55.879 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:56.617 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:56.618 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 13:59:56.856 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:57.262 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 13:59:57.393 INFO 11 --- [pool-ocp-schedules-16,,4ba6a6e4682a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 13:59:58.103 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5aac0ea281874ed2,d3d7577c0791] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=5aac0ea281874ed2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 13:59:58.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5aac0ea281874ed2,d3d7577c0791] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=5aac0ea281874ed2, duration=6 ms] 2023-05-15 13:59:58.257 INFO 11 --- [pool-overview-container-6,,e0233efd94bb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 13:59:58.506 INFO 11 --- [pool-scanner--15968902-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 13:59:59.618 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 13:59:59.619 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:00.000 INFO 11 --- [pool-ocp-schedules-17,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:00:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:00:00.000 INFO 11 --- [pool-ocp-schedules-17,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:00:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130400 2023-05-15 14:00:00.001 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:00:00.001 INFO 11 --- [pool-ocp-schedules-3,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:00:00.013 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:00:00.013 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:00:00.013 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:00:00.013 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:00:00.020 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:00:00.020 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:00:00.020 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:00:00.020 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:00:00.032 INFO 11 --- [pool-ocp-schedules-17,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:00:00.065 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:00:00.074 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:00:00.074 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:00:00.074 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:00:00.074 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:00:00.074 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:00:00.082 WARN 11 --- [pool-ocp-schedules-3,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:00:00.082 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:00:00.617 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:00:00.750 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.750 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=12, name=Refresh agent config, templateName=RefreshAgentConfigSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.752 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Refresh agent config, nodeCount=1, taskCount=1 2023-05-15 14:00:00.753 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.753 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Refresh agent config, taskDefinitionId=12, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.758 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:58:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.758 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 13:58:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.763 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:00:00.763 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.763 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.763 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.768 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:00.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.768 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=15, name=Sync system event, templateName=SyncSystemEventSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-05-15 13:50:00.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.774 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync system event, nodeCount=2, taskCount=3 2023-05-15 14:00:00.774 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.774 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync system event, taskDefinitionId=15, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.786 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-05-15 14:00:00.786 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.787 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.791 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.791 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.821 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:00:00.821 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.821 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.827 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.827 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:59:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.830 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:00:00.830 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.830 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.835 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.835 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:59:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.846 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:00:00.846 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.846 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.850 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.850 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:59:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:00:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.865 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:31.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.865 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 13:59:31.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.868 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:00:00.868 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.868 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.873 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.873 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.876 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-05-15 14:00:00.876 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.876 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.882 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:55:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.882 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 13:55:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.913 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131826, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:00:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.915 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131826, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:00:00.911+08:00, endTime=null} 2023-05-15 14:00:00.919 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:01.0, nextRun=2023-05-15 14:00:00.0, currentDate=Mon May 15 14:00:00 CST 2023 2023-05-15 14:00:00.919 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 13:59:01.0, nextRunTime=2023-05-15 14:00:00.0} 2023-05-15 14:00:00.922 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:00:00.922 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:00.922 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:00.925 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131827, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.926 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131827, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-05-15T14:00:00.923+08:00, endTime=null} 2023-05-15 14:00:00.933 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2aa5e58efe64] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131826, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116234, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.911+08:00, sub_task_instance_id=131826}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:00.933 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2aa5e58efe64] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:00:00.936 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131824, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.938 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131824, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:00:00.934+08:00, endTime=null} 2023-05-15 14:00:00.942 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0309877e1761] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131827, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116235, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.922+08:00, sub_task_instance_id=131827}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:00.942 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0309877e1761] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-05-15 14:00:00.947 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131825, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.948 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131825, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:00:00.944+08:00, endTime=null} 2023-05-15 14:00:00.950 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:00:00.950 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:00:00.951 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:00:00.951 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:00:00.951 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:00:00.951 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:00:00.959 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,782bcd013d3b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131824, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116232, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.934+08:00, sub_task_instance_id=131824}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:00.961 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131815, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.963 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131815, name=Refresh all agent config, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.RefreshAllAgentConfigTask, seriesId=0, startTime=2023-05-15T14:00:00.958+08:00, endTime=null} 2023-05-15 14:00:00.963 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.964 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:00:00.965 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:00:00.965 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.966 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.966 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.966 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.966 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.966 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.967 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.968 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.968 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.968 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.969 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:00:00.969 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.969 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.969 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:00:00.970 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.970 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:00:00.970 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6fc4746c7454] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131825, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116233, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.944+08:00, sub_task_instance_id=131825}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:00.970 INFO 11 --- [pool-3242961-thread-2,fb27d19334114ed7,270f5cdcca55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-3,fb27d19334114ed7,d45933d204fe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:00:00.971 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-4,fb27d19334114ed7,76d109a9a1ee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-1,fb27d19334114ed7,f848b2f8e25d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.972 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.978 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.979 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:00:00.979 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131818, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.979 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.979 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.980 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:00:00.981 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.982 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131818, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-05-15T14:00:00.976+08:00, endTime=null} 2023-05-15 14:00:00.982 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.982 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.982 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,782bcd013d3b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:00:00.983 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:00.983 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,782bcd013d3b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:00:00.983 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:00:00.983 INFO 11 --- [pool-3242961-thread-5,fb27d19334114ed7,22330cdbbbb6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:00:00.985 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:00:00.985 INFO 11 --- [pool-3242961-thread-6,fb27d19334114ed7,b00cab0164c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:00:00.985 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131815, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116226, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.958+08:00, sub_task_instance_id=131815}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:00.986 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2aa5e58efe64] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:00:00.987 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,782bcd013d3b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131824, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:00.990 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2aa5e58efe64] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131826, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:00.990 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:00.990 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.990 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130400000000] 2023-05-15 14:00:00.991 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2023-05-15 14:00:00.992 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2023-05-15 14:00:00.992 INFO 11 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2023-05-15 14:00:00.992 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2023-05-15 14:00:00.992 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2023-05-15 14:00:00.993 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131816, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:00.994 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131816, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:00:00.991+08:00, endTime=null} 2023-05-15 14:00:00.997 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6fc4746c7454] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:00:00.998 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:00.998 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6fc4746c7454] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:00:00.998 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:00.999 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130400000000] 2023-05-15 14:00:01.001 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6fc4746c7454] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:00:01.004 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131817, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:01.006 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131817, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:00:01.002+08:00, endTime=null} 2023-05-15 14:00:01.006 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6fc4746c7454] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131825, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.007 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fc264d6fc4e1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131818, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116228, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.976+08:00, sub_task_instance_id=131818}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.013 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,73e00a59a7f7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131816, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116227, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:00.990+08:00, sub_task_instance_id=131816}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.016 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fc264d6fc4e1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131818, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.022 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131822, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:01.024 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131822, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-05-15T14:00:01.019+08:00, endTime=null} 2023-05-15 14:00:01.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a7e15d13c29] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131817, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116227, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:01.002+08:00, sub_task_instance_id=131817}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.029 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,73e00a59a7f7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131816, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType election_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2023-05-15 14:00:01.038 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:01.040 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a7e15d13c29] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131817, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.042 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131823, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:01.047 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131823, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:00:01.039+08:00, endTime=null} 2023-05-15 14:00:01.051 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:00:01.051 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:00:01.051 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:00:01.051 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2023-05-15 14:00:01.052 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e770f0da374] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131822, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116230, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:01.019+08:00, sub_task_instance_id=131822}, listMap={cluster_ids=[[1,2,1,1]], tenant_ids=[[4,6,2,3]]}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.053 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e770f0da374] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [4, 6, 2, 3] 2023-05-15 14:00:01.061 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2023-05-15 14:00:01.064 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2023-05-15 14:00:01.064 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:01.065 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131821, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:01.068 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,7305e0deee5f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131823, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116231, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:01.038+08:00, sub_task_instance_id=131823}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.068 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131821, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-05-15T14:00:01.058+08:00, endTime=null} 2023-05-15 14:00:01.068 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2023-05-15 14:00:01.068 INFO 11 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2023-05-15 14:00:01.073 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:01.074 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:00:01.075 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:00:01.075 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:00:01.075 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:00:01.075 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:00:01.080 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,7305e0deee5f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131823, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.080 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:01.080 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:01.080 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:01.080 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:01.081 INFO 11 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:01.081 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:01.083 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:01.094 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(agent.http.basic.auth.metricAuthEnabled=true), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}])]), params:null 2023-05-15 14:00:01.099 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,a239c6299af5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131821, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116229, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:01.058+08:00, sub_task_instance_id=131821}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:01.101 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,a239c6299af5] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-05-15 14:00:01.101 INFO 11 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2023-05-15 14:00:01.103 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=ocpcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=1), (monagent.ob.zone.name=zone1), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/admin/ob), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=inactive), (monagent.pipeline.plan.monitor.status=inactive), (monagent.pipeline.sql.plan.status=inactive), (monagent.pipeline.transaction.status=inactive), (monagent.pipeline.session.status=inactive), (monagent.pipeline.slow.sql.status=inactive)]), params:null 2023-05-15 14:00:01.108 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:00:01.108 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2023-05-15 14:00:01.108 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,a239c6299af5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131821, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.112 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone3), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:00:01.113 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:00:01.114 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:00:01.114 INFO 11 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone1), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:00:01.116 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:00:01.117 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone2), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:00:01.119 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2023-05-15 14:00:01.120 INFO 11 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2023-05-15 14:00:01.120 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:01.121 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:01.121 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.121 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:01.123 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:00:01.123 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:00:01.124 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2023-05-15 14:00:01.126 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:00:01.130 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:01.130 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.131 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:00:01.131 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:00:01.132 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.132 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.132 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.133 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:01.134 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:00:01.138 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.138 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.139 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,7e0a39040ffc] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131815, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.139 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:00:01.141 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:00:01.142 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:00:01.142 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:00:01.143 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e770f0da374] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131822, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.149 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116226, status:SUCCESS 2023-05-15 14:00:01.150 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116226, status:SUCCESS 2023-05-15 14:00:01.153 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,f57a64f1ce004990,a4b80b1411ce] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=f57a64f1ce004990, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=ocpcluster,1,false,2,] 2023-05-15 14:00:01.154 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,f57a64f1ce004990,a4b80b1411ce] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=f57a64f1ce004990, duration=2 ms] 2023-05-15 14:00:01.155 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:00:01.156 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:01.156 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.156 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:00:01.160 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116227, status:SUCCESS 2023-05-15 14:00:01.161 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116227, status:SUCCESS 2023-05-15 14:00:01.171 WARN 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obcluster, ob tenant id:1001 2023-05-15 14:00:01.172 INFO 11 --- [pool-3242962-thread-2,fb27d19334114ed7,c60d181f89b1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:00:01.175 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:01.175 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.175 INFO 11 --- [ForkJoinPool-65996-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.10, 2982] 2023-05-15 14:00:01.178 WARN 11 --- [pool-ocp-schedules-3,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:00:01.179 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.159600346S 2023-05-15 14:00:01.179 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:00:01.181 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116229, status:SUCCESS 2023-05-15 14:00:01.182 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116229, status:SUCCESS 2023-05-15 14:00:01.196 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116230, status:SUCCESS 2023-05-15 14:00:01.218 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116230, status:SUCCESS 2023-05-15 14:00:01.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,e7a04844a4164bcc,2b7a6cda2d88] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=e7a04844a4164bcc, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-05-15 14:00:01.224 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,e7a04844a4164bcc,2b7a6cda2d88] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e7a04844a4164bcc, duration=2 ms] 2023-05-15 14:00:01.225 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.226 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.227 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:00:01.233 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116231, status:SUCCESS 2023-05-15 14:00:01.234 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116231, status:SUCCESS 2023-05-15 14:00:01.242 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116232, status:SUCCESS 2023-05-15 14:00:01.243 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116232, status:SUCCESS 2023-05-15 14:00:01.251 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116233, status:SUCCESS 2023-05-15 14:00:01.252 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116233, status:SUCCESS 2023-05-15 14:00:01.260 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116234, status:SUCCESS 2023-05-15 14:00:01.261 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116234, status:SUCCESS 2023-05-15 14:00:01.263 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.263 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.263 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.11, 2982] 2023-05-15 14:00:01.265 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.265 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.265 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.13, 2982] 2023-05-15 14:00:01.267 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:01.267 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:01.267 INFO 11 --- [ForkJoinPool-65996-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.12, 2982] 2023-05-15 14:00:01.280 INFO 11 --- [pool-3242962-thread-1,fb27d19334114ed7,1bf5b3670c8d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:00:01.281 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0309877e1761] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-05-15 14:00:01.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0309877e1761] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131827, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:01.348 INFO 11 --- [pool-ocp-schedules-3,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:00:01.402 INFO 11 --- [pool-ocp-schedules-3,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:00:01.441 INFO 11 --- [pool-ocp-schedules-3,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:00:01.452 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:00:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,b135b115d9ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130460, interval=60 2023-05-15 14:00:01.518 INFO 11 --- [pool-prometheus-minute-schedule-1,,e05943e85c01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130460, interval=60 2023-05-15 14:00:01.539 INFO 11 --- [pool-prometheus-minute-schedule-2,,b135b115d9ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:00:01.539 INFO 11 --- [pool-prometheus-minute-schedule-1,,e05943e85c01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:00:01.539 INFO 11 --- [pool-prometheus-minute-schedule-2,,b135b115d9ef] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130401, count=8, elapsed=1682446271138 2023-05-15 14:00:01.539 INFO 11 --- [pool-prometheus-minute-schedule-1,,e05943e85c01] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130401, count=6, elapsed=1682446271138 2023-05-15 14:00:01.644 INFO 11 --- [pool-ocp-schedules-3,,] o.s.boot.SpringApplication : Started application in 1.631 seconds (JVM running for 600057.618) 2023-05-15 14:00:01.644 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.631589251S 2023-05-15 14:00:01.884 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:00:01.884 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:00:01.884 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:00:01.884 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:00:02.203 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131828, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:02.209 INFO 11 --- [pool-ocp-schedules-3,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:00:02.226 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131828, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-05-15T14:00:02.185+08:00, endTime=null} 2023-05-15 14:00:02.257 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131828, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116236, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:02.185+08:00, sub_task_instance_id=131828}, listMap={cluster_ids=[[2,1,1,1]], tenant_ids=[[6,4,3,2]]}}, executor=10.xxx.xx.10 2023-05-15 14:00:02.257 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[6, 4, 3, 2] 2023-05-15 14:00:02.265 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1684130402257000 2023-05-15 14:00:02.265 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1684130101676000 2023-05-15 14:00:02.268 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131829, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:02.271 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131829, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:00:02.264+08:00, endTime=null} 2023-05-15 14:00:02.285 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2023-05-15 14:00:02.291 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1684130402287000 2023-05-15 14:00:02.291 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1684130101732000 2023-05-15 14:00:02.294 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131819, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:02.295 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,bf517a5b26f5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131829, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116237, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:02.264+08:00, sub_task_instance_id=131829}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:02.302 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131819, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=1, startTime=2023-05-15T14:00:02.286+08:00, endTime=null} 2023-05-15 14:00:02.306 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2023-05-15 14:00:02.316 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1684130402308000 2023-05-15 14:00:02.316 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1684130101663000 2023-05-15 14:00:02.327 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,bf517a5b26f5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131829, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:02.331 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-05-15 14:00:02.336 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1684130402332000 2023-05-15 14:00:02.336 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1684130101715000 2023-05-15 14:00:02.347 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-05-15 14:00:02.348 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131820, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:02.352 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131820, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=2, startTime=2023-05-15T14:00:02.323+08:00, endTime=null} 2023-05-15 14:00:02.356 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a6abc05e82fa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131828, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:02.365 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131819, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116228, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:02.286+08:00, sub_task_instance_id=131819}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:02.373 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116235, status:SUCCESS 2023-05-15 14:00:02.374 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116235, status:SUCCESS 2023-05-15 14:00:02.378 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131820, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116228, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:02.318+08:00, sub_task_instance_id=131820}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:02.381 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116236, status:SUCCESS 2023-05-15 14:00:02.382 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116236, status:SUCCESS 2023-05-15 14:00:02.389 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116237, status:SUCCESS 2023-05-15 14:00:02.390 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116237, status:SUCCESS 2023-05-15 14:00:02.394 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:02.394 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:02.394 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.395 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:02.399 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:02.399 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.400 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-05-15 08:00:12.528118, 2023-05-15 14:00:02.372, 1000] 2023-05-15 14:00:02.400 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2023-05-15 08:00:12.528118, endTime 2023-05-15 14:00:02.372 2023-05-15 14:00:02.401 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.402 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.402 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.403 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:02.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d866b0fc52b9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131819, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:02.408 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.408 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.408 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-05-15 13:40:31.376107, 2023-05-15 14:00:02.381, 1000] 2023-05-15 14:00:02.466 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 52 system events to database for cluster 2 2023-05-15 14:00:02.466 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 52 system events for cluster 2, startTime 2023-05-15 13:40:31.376107, endTime 2023-05-15 14:00:02.381, limit 1000, offset 0 2023-05-15 14:00:02.488 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:02.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.490 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:02.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:02.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:02.577 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2023-05-15 13:40:31.376107, 2023-05-15 14:00:02.381, 1000, 1000] 2023-05-15 14:00:02.578 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2023-05-15 13:40:31.376107, endTime 2023-05-15 14:00:02.381 2023-05-15 14:00:02.585 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,9738d8dde59d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131820, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:02.619 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:02.620 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:02.621 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,c8a8a3ee1d114c1e,2fc24e12136b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=c8a8a3ee1d114c1e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:02.624 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,c8a8a3ee1d114c1e,2fc24e12136b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=c8a8a3ee1d114c1e, duration=4 ms] 2023-05-15 14:00:03.423 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116228, status:SUCCESS 2023-05-15 14:00:03.424 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116228, status:SUCCESS 2023-05-15 14:00:04.002 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:50:05.0, nextRun=2023-05-15 14:00:04.0, currentDate=Mon May 15 14:00:04 CST 2023 2023-05-15 14:00:04.003 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2023-05-15 13:50:05.0, nextRunTime=2023-05-15 14:00:04.0} 2023-05-15 14:00:04.006 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2023-05-15 14:00:04.007 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:04.007 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Partition rollover daemon, taskDefinitionId=1, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:04.394 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131830, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:04.396 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131830, name=Partition rollover daemon, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=2023-05-15T14:00:04.392+08:00, endTime=null} 2023-05-15 14:00:04.413 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131830, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116238, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:04.391+08:00, sub_task_instance_id=131830}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:04.417 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2023-05-15 14:00:04.417 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2023-05-15 14:00:04.419 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.419 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.420 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.421 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.421 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.422 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2023-05-15 14:00:04.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.458 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.458 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2023-05-15 14:00:04.479 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.479 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.480 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.481 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.481 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.481 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2023-05-15 14:00:04.503 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.503 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.504 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.504 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.504 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.505 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2023-05-15 14:00:04.530 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.530 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.531 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.531 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.531 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.532 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2023-05-15 14:00:04.538 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.538 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.539 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.539 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.539 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.539 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2023-05-15 14:00:04.545 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.546 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.546 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.546 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.546 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.547 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2023-05-15 14:00:04.560 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.560 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.562 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2023-05-15 14:00:04.570 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.570 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.574 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.575 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.575 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.575 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2023-05-15 14:00:04.581 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.581 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.581 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.582 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.582 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.582 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2023-05-15 14:00:04.591 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.591 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.592 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.593 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.593 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.593 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2023-05-15 14:00:04.608 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.608 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.608 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.609 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.609 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.609 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2023-05-15 14:00:04.614 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.614 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.614 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.615 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.615 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.615 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2023-05-15 14:00:04.622 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.622 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.622 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.623 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.623 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.623 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2023-05-15 14:00:04.628 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.629 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.629 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.630 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.630 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.630 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2023-05-15 14:00:04.634 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.635 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.635 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.636 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.636 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.636 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2023-05-15 14:00:04.640 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.640 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.641 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.641 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.641 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.642 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2023-05-15 14:00:04.646 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.646 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.648 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2023-05-15 14:00:04.656 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.656 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.657 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.657 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.657 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.658 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2023-05-15 14:00:04.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.663 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2023-05-15 14:00:04.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.668 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2023-05-15 14:00:04.671 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.671 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.671 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.672 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.672 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.672 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2023-05-15 14:00:04.676 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.676 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.676 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.676 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.677 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.677 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2023-05-15 14:00:04.680 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.680 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.681 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.681 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.681 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.681 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2023-05-15 14:00:04.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.687 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.687 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.688 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2023-05-15 14:00:04.693 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.693 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.693 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.694 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.694 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.694 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2023-05-15 14:00:04.700 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.700 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.701 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.701 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.702 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.702 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2023-05-15 14:00:04.710 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.710 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.710 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.711 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.711 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.712 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2023-05-15 14:00:04.714 INFO 11 --- [pool-ocp-schedules-15,,6da65ca2be3a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:00:04.719 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.719 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.720 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.720 INFO 11 --- [pool-ocp-schedules-15,,6da65ca2be3a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:00:04.721 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.721 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.721 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2023-05-15 14:00:04.728 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.728 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.729 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2023-05-15 14:00:04.738 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.738 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.739 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2023-05-15 14:00:04.748 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.748 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.749 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.749 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.749 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.750 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2023-05-15 14:00:04.756 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.757 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.757 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.758 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.758 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.758 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2023-05-15 14:00:04.773 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.773 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.773 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.774 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.774 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.775 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat 2023-05-15 14:00:04.783 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.783 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.783 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.784 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.784 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.784 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2023-05-15 14:00:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:04.789 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.789 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.790 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.790 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.791 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.791 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2023-05-15 14:00:04.795 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.795 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.796 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.796 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.796 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.797 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2023-05-15 14:00:04.802 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.802 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.803 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.803 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.803 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.804 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2023-05-15 14:00:04.809 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.809 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.809 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.810 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.810 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.811 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2023-05-15 14:00:04.815 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.815 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.815 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.816 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.816 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.817 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2023-05-15 14:00:04.831 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.831 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:04.830869+08:00, lastProcessTime=2023-05-15T13:59:54.831489+08:00 2023-05-15 14:00:04.832 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.833 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.833 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.833 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2023-05-15 14:00:04.853 INFO 11 --- [alarm-detect-14,,4ed9848db59c] c.o.o.s.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, activeAt=2023-05-15T13:59:34.844+08:00 2023-05-15 14:00:04.853 INFO 11 --- [alarm-detect-14,,4ed9848db59c] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType tenant_cpu_percent_over_threshold targets.size 10 was marked as not match 2023-05-15 14:00:04.859 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.859 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.860 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.861 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.861 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.862 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2023-05-15 14:00:04.865 INFO 11 --- [alarm-detect-4,,889dfdc832bc] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130384 ~ 1684130394 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.677, values={1684130390=245.677}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:04.870 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.870 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.870 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.871 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.871 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.871 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2023-05-15 14:00:04.877 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.877 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.877 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.877 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.877 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.878 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.881 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2023-05-15 14:00:04.885 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.885 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.885 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.885 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.885 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.886 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2023-05-15 14:00:04.889 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2023-05-15 14:00:04.889 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2023-05-15 14:00:04.890 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.890 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.890 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.891 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.891 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.891 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2023-05-15 14:00:04.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:04.830869+08:00 2023-05-15 14:00:04.902 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.903 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.903 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:00:04.904 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:00:04.904 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:04.904 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2023-05-15 14:00:04.908 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2023-05-15 14:00:04.916 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6671db10cac6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131830, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:05.046 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:05.0, nextRun=2023-05-15 14:00:05.0, currentDate=Mon May 15 14:00:05 CST 2023 2023-05-15 14:00:05.047 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 13:59:05.0, nextRunTime=2023-05-15 14:00:05.0} 2023-05-15 14:00:05.051 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:00:05.051 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:05.051 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:05.421 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131831, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:05.423 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131831, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:00:05.418+08:00, endTime=null} 2023-05-15 14:00:05.445 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d255d62a9c9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131831, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116239, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:05.418+08:00, sub_task_instance_id=131831}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:05.472 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,2d0348d3cb00] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:00:05.472 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,804076637af3] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:00:05.475 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,959861267efa] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:00:05.476 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116238, status:SUCCESS 2023-05-15 14:00:05.477 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116238, status:SUCCESS 2023-05-15 14:00:05.520 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,2d0348d3cb00] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:00:05.521 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,804076637af3] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:00:05.526 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,959861267efa] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:00:05.548 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d255d62a9c9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131831, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:05.620 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:05.621 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:05.871 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:05.873 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:05.879 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:05.881 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:06.508 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116239, status:SUCCESS 2023-05-15 14:00:06.510 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116239, status:SUCCESS 2023-05-15 14:00:06.827 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,8cdb0142b7f74fe9,52f902a0e557] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8cdb0142b7f74fe9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:06.832 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,8cdb0142b7f74fe9,52f902a0e557] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8cdb0142b7f74fe9, duration=6 ms] 2023-05-15 14:00:06.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:07.414 INFO 11 --- [pool-ocp-schedules-17,,1672d29a5728] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:08.259 INFO 11 --- [pool-overview-container-3,,2924263b4d58] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:00:08.259 INFO 11 --- [pool-overview-container-3,,2924263b4d58] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:08.511 INFO 11 --- [pool-scanner--15968995-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:00:08.621 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:08.622 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:10.049 INFO 11 --- [pool-ocp-schedules-9,,7431deeeb8ba] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:00:10.123 INFO 11 --- [pool-ocp-schedules-2,,ee42c21463cc] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:00:10.124 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:00:10.124 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:00:10.124 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:00:10.124 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:00:10.124 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:00:10.130 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:10.130 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:10.130 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:10.130 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:10.130 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:10.157 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:00:10.157 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:00:10.158 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:00:10.159 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:00:10.161 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:00:10.159305323+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:00:10.160277489+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:00:10.160775964+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:00:10.159408384+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:00:10.162+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:00:10.162+08:00 2023-05-15 14:00:10.162 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:00:10.162+08:00 2023-05-15 14:00:10.163 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:00:10.162+08:00 2023-05-15 14:00:10.163 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:00:10.162668006+08:00 2023-05-15 14:00:10.163 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:00:10.163+08:00 2023-05-15 14:00:10.172 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:00:10.162+08:00 and status AVAILABLE 2023-05-15 14:00:10.179 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:00:10.202 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130410 0 0 , error= 2023-05-15 14:00:10.202 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:00:10.162+08:00 and status AVAILABLE 2023-05-15 14:00:10.203 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:00:10.202+08:00 2023-05-15 14:00:10.205 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:00:10.162+08:00 and status AVAILABLE 2023-05-15 14:00:10.207 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:00:10.202+08:00 and status AVAILABLE 2023-05-15 14:00:10.208 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:00:10.163+08:00 and status AVAILABLE 2023-05-15 14:00:10.210 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:00:10.210 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:00:10.211 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:00:10.162+08:00 and status AVAILABLE 2023-05-15 14:00:10.212 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:00:10.231 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130410 0 0 , error= 2023-05-15 14:00:10.231 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:00:10.231+08:00 2023-05-15 14:00:10.232 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:00:10.235 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:00:10.251 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130410 0 0 , error= 2023-05-15 14:00:10.252 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:00:10.252+08:00 2023-05-15 14:00:10.252 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:00:10.231+08:00 and status AVAILABLE 2023-05-15 14:00:10.253 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:10.267 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:00:10.271 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:00:10.252+08:00 and status AVAILABLE 2023-05-15 14:00:10.274 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:00:10.282 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:10.284 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:10.305 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:00:10.309 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:00:10.313 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:00:10.360 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:00:10.362 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:00:10.374 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:00:10.376 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:00:10.397 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:00:10.406 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:00:10.408 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:00:10.417 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:00:10.432 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:00:10.456 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:00:10.458 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:00:10.469 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:00:10.471 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:00:10.486 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:00:10.489 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:00:10.513 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:00:10.527 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:00:10.536 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:00:10.542 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:00:10.547 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:00:10.550 INFO 11 --- [pool-Check host status29,,7589d22bd586] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:00:10.558 INFO 11 --- [pool-Check host status21,,da8762a436da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:00:10.633 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:00:10.646 INFO 11 --- [pool-Check host status6,,496302f33694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:00:11.175 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130411 0 0 , error= 2023-05-15 14:00:11.176 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:00:11.176+08:00 2023-05-15 14:00:11.185 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:00:11.176+08:00 and status AVAILABLE 2023-05-15 14:00:11.187 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:00:11.214 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130411 0 0 , error= 2023-05-15 14:00:11.214 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:00:11.214+08:00 2023-05-15 14:00:11.218 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:11.220 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:00:11.214+08:00 and status AVAILABLE 2023-05-15 14:00:11.222 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:00:11.233 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:11.249 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:00:11.259 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:00:11.301 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:00:11.318 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:00:11.320 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:00:11.335 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:00:11.343 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:00:11.357 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:00:11.389 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:00:11.404 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:00:11.416 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:00:11.427 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:00:11.430 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:00:11.461 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:00:11.480 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:00:11.504 INFO 11 --- [pool-Check host status4,,251e1f3d63bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:00:11.520 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:00:11.531 INFO 11 --- [pool-Check host status7,,5f50b1317138] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:00:11.548 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:00:11.549 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:00:11.549 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:00:11.549 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:00:11.550 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:00:11.554 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:11.554 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:11.556 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:11.562 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:11.567 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:11.575 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:00:11.576 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:00:11.576 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:00:11.581 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:00:11.586 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:00:11.593 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:00:11.594 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:00:11.594 INFO 11 --- [pool-Check host status23,,c3cad268fc4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:00:11.596 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:00:11.596 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:00:11.596 INFO 11 --- [pool-Check host status11,,5181756bc2ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:00:11.597 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:00:11.597 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:00:11.597 INFO 11 --- [pool-Check host status20,,332e02802c49] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:00:11.601 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:00:11.601 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:00:11.601 INFO 11 --- [pool-Check host status27,,22ccfe4efc42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:00:11.622 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:11.623 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:11.626 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:00:11.626 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:00:11.626 INFO 11 --- [pool-Check host status5,,dbec04bfb27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:00:11.712 INFO 11 --- [pool-ocp-schedules-8,,b030204a6281] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:11.712 INFO 11 --- [pool-ocp-schedules-8,,b030204a6281] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:12.264 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:14.245 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130340, nodeCount=327, affectRows=327 2023-05-15 14:00:14.246 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130340, nodeCount=690, affectRows=690 2023-05-15 14:00:14.247 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130340, nodeCount=344, affectRows=344 2023-05-15 14:00:14.247 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130340, nodeCount=329, affectRows=329 2023-05-15 14:00:14.247 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130340, nodeCount=1000, affectRows=1000 2023-05-15 14:00:14.249 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130340, nodeCount=344, validCount=344 2023-05-15 14:00:14.249 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130340, nodeCount=327, validCount=327 2023-05-15 14:00:14.250 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130340, nodeCount=329, validCount=329 2023-05-15 14:00:14.252 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130340, nodeCount=690, validCount=690 2023-05-15 14:00:14.253 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130340, nodeCount=1000, validCount=1000 2023-05-15 14:00:14.623 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:14.625 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:14.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:14.829037+08:00, lastProcessTime=2023-05-15T14:00:04.830869+08:00 2023-05-15 14:00:14.869 INFO 11 --- [alarm-detect-2,,0e279a1732bc] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130394 ~ 1684130404 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.677, values={1684130400=245.677}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:14.918 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:14.829037+08:00 2023-05-15 14:00:15.873 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:15.875 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:15.880 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:15.883 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:16.868 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:17.435 INFO 11 --- [pool-ocp-schedules-9,,6e2ed1796dd3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:17.625 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:17.626 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:18.247 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,93906b84fb6c4bb5,a63fa5d57417] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=93906b84fb6c4bb5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:18.259 INFO 11 --- [pool-overview-container-4,,d4491f834984] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:18.261 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,93906b84fb6c4bb5,a63fa5d57417] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=93906b84fb6c4bb5, duration=14 ms] 2023-05-15 14:00:18.504 INFO 11 --- [pool-scanner--15969095-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:20.502 INFO 11 --- [pool-scanner--15969096-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:20.530 INFO 11 --- [pool-scanner--15969097-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:20.546 INFO 11 --- [pool-scanner--15969098-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:20.579 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:21.0, nextRun=2023-05-15 14:00:20.0, currentDate=Mon May 15 14:00:20 CST 2023 2023-05-15 14:00:20.580 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 13:59:21.0, nextRunTime=2023-05-15 14:00:20.0} 2023-05-15 14:00:20.584 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:00:20.584 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:20.584 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:20.589 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.597 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.626 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:20.627 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:20.643 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.648 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.652 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131832, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:20.655 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131832, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:00:20.649+08:00, endTime=null} 2023-05-15 14:00:20.669 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.674 INFO 11 --- [pool-ocp-schedules-10,,65469d6dc755] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:20.689 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,7a90fbc80f8d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131832, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116240, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:20.649+08:00, sub_task_instance_id=131832}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:20.691 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,7a90fbc80f8d] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:00:20.692 INFO 11 --- [ForkJoinPool-65999-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:00:20.692 INFO 11 --- [ForkJoinPool-65999-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:00:20.696 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,7a90fbc80f8d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131832, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:20.787 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116240, status:SUCCESS 2023-05-15 14:00:20.787 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116240, status:SUCCESS 2023-05-15 14:00:21.207 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b7fd049a678440c2,257f0d29c160] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=b7fd049a678440c2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:00:21.208 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b7fd049a678440c2,257f0d29c160] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=b7fd049a678440c2, duration=2 ms] 2023-05-15 14:00:21.602 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-05-15 14:00:21.626 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:00:21.650 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:00:21.676 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:00:21.697 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:00:21.718 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:00:21.739 INFO 11 --- [pool-ocp-schedules-20,,03780e069a5e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=5 2023-05-15 14:00:21.788 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:21.790 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:21.792 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:23.627 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:23.628 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:24.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:24.831676+08:00, lastProcessTime=2023-05-15T14:00:14.829037+08:00 2023-05-15 14:00:24.863 INFO 11 --- [alarm-detect-4,,42e19dfb8d03] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130354 ~ 1684130414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388797.988, values={1684130400=388797.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:00:24.864 INFO 11 --- [alarm-detect-4,,42e19dfb8d03] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130354 ~ 1684130414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993595.358, values={1684130400=993595.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:24.865 INFO 11 --- [alarm-detect-7,,f275738cc217] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130354 ~ 1684130414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388797.938, values={1684130400=388797.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:00:24.865 INFO 11 --- [alarm-detect-13,,729935d25857] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130404 ~ 1684130414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.677, values={1684130410=245.677}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:24.865 INFO 11 --- [alarm-detect-7,,f275738cc217] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130354 ~ 1684130414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993595.107, values={1684130400=993595.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:24.867 INFO 11 --- [alarm-detect-4,,42e19dfb8d03] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:00:24.868 INFO 11 --- [alarm-detect-7,,f275738cc217] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:00:24.868 INFO 11 --- [alarm-detect-13,,729935d25857] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:00:24.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:24.831676+08:00 2023-05-15 14:00:25.742 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:26.0, nextRun=2023-05-15 14:00:25.0, currentDate=Mon May 15 14:00:25 CST 2023 2023-05-15 14:00:25.742 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 13:59:26.0, nextRunTime=2023-05-15 14:00:25.0} 2023-05-15 14:00:25.754 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:00:25.754 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:25.754 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:25.876 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:25.878 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:25.883 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:25.885 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:26.407 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,348fd331ad3a479e,886ca53d00f5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=348fd331ad3a479e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:26.413 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,348fd331ad3a479e,886ca53d00f5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=348fd331ad3a479e, duration=7 ms] 2023-05-15 14:00:26.629 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:26.630 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:26.759 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131833, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:26.761 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131833, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:00:26.756+08:00, endTime=null} 2023-05-15 14:00:26.787 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131833, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116241, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:26.756+08:00, sub_task_instance_id=131833}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:00:26.829 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:00:26.830 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:00:26.831 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:00:26.831 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:00:26.832 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:00:26.838 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57d937bbbc13] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131833, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:26.871 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:26.916 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116241, status:SUCCESS 2023-05-15 14:00:26.918 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116241, status:SUCCESS 2023-05-15 14:00:27.270 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:27.390 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c8a43c6baffb4fd2,500d2d0126a4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=c8a43c6baffb4fd2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:27.396 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c8a43c6baffb4fd2,500d2d0126a4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=c8a43c6baffb4fd2, duration=6 ms] 2023-05-15 14:00:27.456 INFO 11 --- [pool-ocp-schedules-19,,eb90e823073c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:28.259 INFO 11 --- [pool-overview-container-1,,05543045df0f] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:00:28.259 INFO 11 --- [pool-overview-container-1,,05543045df0f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:28.506 INFO 11 --- [pool-scanner--15969199-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:29.630 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:29.631 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:30.061 INFO 11 --- [pool-ocp-schedules-13,,59a5eb64ae16] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:00:30.160 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.196 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.198 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.199 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.201 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.202 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.204 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.205 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.206 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.208 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.210 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.212 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.214 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.226 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.232 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.237 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.244 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.246 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.249 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.251 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.252 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.257 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.262 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.266 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.271 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:54:30.031, end=2023-05-15T13:59:30.156} 2023-05-15 14:00:30.271 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:00:30.906 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:59:31.0, nextRun=2023-05-15 14:00:30.0, currentDate=Mon May 15 14:00:30 CST 2023 2023-05-15 14:00:30.906 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 13:59:31.0, nextRunTime=2023-05-15 14:00:30.0} 2023-05-15 14:00:30.915 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:00:30.915 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:30.915 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:30.915 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:30.929 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:00:30.0, currentDate=Mon May 15 14:00:30 CST 2023 2023-05-15 14:00:30.929 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:00:30.0} 2023-05-15 14:00:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:00:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:00:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:00:31.720 INFO 11 --- [pool-ocp-schedules-19,,21e74cba1d28] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:31.720 INFO 11 --- [pool-ocp-schedules-19,,21e74cba1d28] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:31.848 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131834, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:31.849 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131834, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:00:31.845+08:00, endTime=null} 2023-05-15 14:00:31.861 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131835, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:31.863 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131835, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:00:31.858+08:00, endTime=null} 2023-05-15 14:00:31.873 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131834, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116242, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:31.845+08:00, sub_task_instance_id=131834}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:31.875 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131836, operation:EXECUTE, state: RUNNING 2023-05-15 14:00:31.876 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131836, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:00:31.872+08:00, endTime=null} 2023-05-15 14:00:31.888 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131835, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116242, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:31.858+08:00, sub_task_instance_id=131835}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:00:31.897 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,980987b98449] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131836, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116243, task_operation=execute, latest_execution_start_time=2023-05-15T14:00:31.872+08:00, sub_task_instance_id=131836}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:00:31.897 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,980987b98449] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:00:31.923 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:00:31.924 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:00:31.924 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:00:31.924 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:00:31.924 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:00:31.924 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:00:31.929 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.930 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.930 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.930 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.937 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.938 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.938 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.938 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.938 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.939 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.939 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.939 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.939 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:00:31.939 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.940 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.941 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.942 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.943 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.943 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.943 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.944 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.945 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.945 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.945 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.953 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.953 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.954 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:00:31.955 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.955 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.955 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:00:31.958 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:00:31.958 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:00:31.960 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.960 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.960 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.961 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.961 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.961 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.961 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.973 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.973 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-2,fb27d19334114ed7,6a19b6b6cf79] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:00:31.973 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:00:31.974 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:00:31.974 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:00:31.974 INFO 11 --- [pool-3243122-thread-1,fb27d19334114ed7,3297a8ba1212] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:00:31.977 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:00:31.977 INFO 11 --- [pool-ocp-schedules-20,,eb6c1f7a35fd] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:00:31.977 INFO 11 --- [pool-3243122-thread-4,fb27d19334114ed7,0b6109a9b640] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:00:31.977 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:00:31.978 INFO 11 --- [pool-3243122-thread-3,fb27d19334114ed7,82990c92fcd9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:00:31.979 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.979 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.980 INFO 11 --- [pool-ocp-schedules-20,,eb6c1f7a35fd] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:00:31.980 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.980 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.980 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.980 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.980 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:31.980 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:00:31.981 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:31.981 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.981 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:31.981 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:00:31.985 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:00:31.985 INFO 11 --- [pool-3243122-thread-6,fb27d19334114ed7,a1cbd1e0f07e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:00:31.987 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:00:31.987 INFO 11 --- [pool-3243122-thread-5,fb27d19334114ed7,55f4c8ce9502] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:00:31.988 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,980987b98449] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:00:31.990 INFO 11 --- [pool-ocp-schedules-10,,db3193ebab63] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:00:31.993 INFO 11 --- [pool-ocp-schedules-10,,db3193ebab63] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:00:31.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,980987b98449] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131836, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:31.996 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.997 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.997 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:31.998 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.998 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:31.998 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:31.998 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:31.998 INFO 11 --- [pool-ocp-schedules-11,,273401a5cda0] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:00:32.000 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.000 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.000 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.000 INFO 11 --- [pool-ocp-schedules-11,,273401a5cda0] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:00:32.001 INFO 11 --- [ForkJoinPool-66000-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:00:32.002 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.002 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.003 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:32.006 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.006 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.006 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.006 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.011 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.011 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.012 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.014 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.014 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.015 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:32.015 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.015 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.015 INFO 11 --- [ForkJoinPool-66000-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:00:32.016 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:32.021 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116243, status:SUCCESS 2023-05-15 14:00:32.022 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116243, status:SUCCESS 2023-05-15 14:00:32.025 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.025 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.025 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:00:32.027 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.028 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.028 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.028 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.029 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.029 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.030 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:32.046 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.046 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.047 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:00:32.047 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.047 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.047 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:00:32.054 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:00:32.061 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.061 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.061 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:00:32.072 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.075 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.076 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.076 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:00:32.083 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.083 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.088 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.089 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:00:32.094 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.098 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.098 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.102 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:00:32.102 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.102 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.103 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.103 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.114 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.115 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.115 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.115 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.115 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.118 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.118 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.121 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.122 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.122 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.122 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.123 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.123 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.123 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.123 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.125 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.125 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.125 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:00:32.126 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.127 WARN 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:00:32.133 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.133 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.134 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:00:32.135 INFO 11 --- [ForkJoinPool-66002-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:00:32.135 INFO 11 --- [ForkJoinPool-66002-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:00:32.135 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:00:32.145 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.146 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.146 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.146 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.147 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.147 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.148 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.149 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.149 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.150 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.150 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.150 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.150 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.152 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.152 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:00:32.154 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.154 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.155 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.175 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.176 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.176 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.176 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.176 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.177 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.178 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.178 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.178 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.190 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.191 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.191 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.191 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.191 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.192 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.192 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.192 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.193 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.193 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.193 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.193 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.193 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.193 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.194 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.194 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.194 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.194 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.195 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.196 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.196 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.196 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.196 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.197 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.197 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.197 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.198 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.198 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.198 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:00:32.199 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.199 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.200 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.202 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.202 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.203 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:00:32.205 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:00:32.206 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:00:32.207 INFO 11 --- [ForkJoinPool-66002-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:00:32.213 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.213 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.214 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.214 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.214 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.215 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.215 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.216 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.217 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.218 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.220 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.220 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.221 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.221 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.222 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:00:32.225 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.226 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.226 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.227 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:00:32.228 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.228 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.228 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.228 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.229 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.230 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.230 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.230 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.236 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.236 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:00:32.238 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:00:32.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.249 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.249 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.249 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.250 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.250 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.250 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:00:32.252 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.252 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:00:32.263 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.265 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.266 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.266 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.266 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:00:32.267 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:00:32.268 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.268 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:00:32.271 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:00:32.273 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.274 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.275 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.275 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.276 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.282 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.282 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.284 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.284 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:32.284 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:00:32.290 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.292 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.294 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.294 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.295 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:32.296 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.297 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:00:32.298 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:00:32.299 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.299 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.300 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:00:32.304 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.306 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:00:32.306 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.306 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.306 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:00:32.308 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:00:32.309 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.310 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.310 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.310 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.314 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:00:32.314 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.315 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:00:32.316 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.318 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:00:32.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:00:32.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:00:32.319 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:00:32.319 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.319 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.320 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:00:32.322 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b83068da1c72] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131835, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:32.328 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.330 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:00:32.330 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:00:32.330 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:00:32.330 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:00:32.333 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:00:32.333 INFO 11 --- [ForkJoinPool-66002-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:00:32.339 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:00:32.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:00:32.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:00:32.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:00:32.346 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,b726822dc014] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131834, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:00:32.631 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:32.633 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:32.973 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/compaction?id=2&tenantId=6, client=192.168.80.229, traceId=4662df97f2584b85, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantCompactionController.getTenantCompaction(Long,Long), args=2,6,] 2023-05-15 14:00:32.999 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:00:32.999 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:00:33.014 ERROR 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526895) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=526895) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:00:33.017 ERROR 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] com.alibaba.druid.pool.DruidDataSource : {dataSource-92} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526895) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=526895) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:00:33.017 INFO 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] com.alibaba.druid.pool.DruidDataSource : {dataSource-92,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:00:33.017 INFO 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] com.alibaba.druid.pool.DruidDataSource : {dataSource-92} closing ... 2023-05-15 14:00:33.021 INFO 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] com.alibaba.druid.pool.DruidDataSource : {dataSource-92} closed 2023-05-15 14:00:33.021 ERROR 11 --- [obsdk-init-thread-9,4662df97f2584b85,fc0eb7f17895] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=526895) Get Location Cache Fail 2023-05-15 14:00:33.022 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526895) Get Location Cache Fail 2023-05-15 14:00:33.022 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526895) Get Location Cache Fail 2023-05-15 14:00:33.023 WARN 11 --- [http-nio-0.0.0.0-8080-exec-49,4662df97f2584b85,844d20b31cc2] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/compaction client=192.168.80.229, traceId=4662df97f2584b85, duration=53 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:00:33.031 ERROR 11 --- [Druid-ConnectionPool-Create-873820103,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526898) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=526898) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:00:33.052 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116242, status:SUCCESS 2023-05-15 14:00:33.053 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116242, status:SUCCESS 2023-05-15 14:00:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:34.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:34.830698+08:00, lastProcessTime=2023-05-15T14:00:24.831676+08:00 2023-05-15 14:00:34.868 INFO 11 --- [alarm-detect-10,,533fc6f339b9] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130414 ~ 1684130424 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130420=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:34.907 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:34.830698+08:00 2023-05-15 14:00:35.633 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:35.634 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:35.877 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:35.880 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:35.885 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:35.886 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:36.874 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:37.489 INFO 11 --- [pool-ocp-schedules-11,,ccb3f36c9e2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:38.258 INFO 11 --- [pool-overview-container-5,,b2986ed7f008] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:38.506 INFO 11 --- [pool-scanner--15969307-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:38.634 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:38.635 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:40.724 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,767c24d1bb284a38,fbf881628e29] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=767c24d1bb284a38, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:40.730 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,767c24d1bb284a38,fbf881628e29] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=767c24d1bb284a38, duration=6 ms] 2023-05-15 14:00:41.636 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:41.637 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:42.272 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:44.637 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:44.638 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:44.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:44.830442+08:00, lastProcessTime=2023-05-15T14:00:34.830698+08:00 2023-05-15 14:00:44.861 INFO 11 --- [alarm-detect-6,,504426e2b02b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130424 ~ 1684130434 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130430=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:44.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:44.830442+08:00 2023-05-15 14:00:45.463 INFO 11 --- [pool-ocp-schedules-18,,300894ec0a3d] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:00:45.474 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,9d860983b74c49be,4a95bab8a4a8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=9d860983b74c49be, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:45.479 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,9d860983b74c49be,4a95bab8a4a8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=9d860983b74c49be, duration=6 ms] 2023-05-15 14:00:45.880 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:45.899 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:45.905 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:45.906 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:46.513 INFO 11 --- [pool-prometheus-hour-schedule-2,,0a57b75fc5c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684134000, interval=3600 2023-05-15 14:00:46.514 INFO 11 --- [pool-prometheus-hour-schedule-1,,92dc43a30614] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684134000, interval=3600 2023-05-15 14:00:46.524 INFO 11 --- [pool-prometheus-hour-schedule-1,,92dc43a30614] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130446, count=0, elapsed=1682446316078 2023-05-15 14:00:46.524 INFO 11 --- [pool-prometheus-hour-schedule-2,,0a57b75fc5c6] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130446, count=0, elapsed=1682446316078 2023-05-15 14:00:46.746 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743} 2023-05-15 14:00:46.775 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.775 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.777 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.777 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.784 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.784 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.785 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.786 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.787 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.787 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.793 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:56:46.689, end=2023-05-15T13:58:46.743},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:00:46.793 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:00:46.793 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:00:46.882 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:47.510 INFO 11 --- [pool-ocp-schedules-18,,597af5f50310] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:47.638 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:47.639 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:48.261 INFO 11 --- [pool-overview-container-3,,fb5fc8c903ef] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:48.507 INFO 11 --- [pool-scanner--15969407-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:50.639 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:50.640 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:51.726 INFO 11 --- [pool-ocp-schedules-1,,f8af9af871b5] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:51.726 INFO 11 --- [pool-ocp-schedules-1,,f8af9af871b5] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:00:51.790 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:51.793 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:51.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,643ff5542e41] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":77297} 2023-05-15 14:00:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,643ff5542e41] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:00:53.640 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:53.641 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:00:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:00:54.831938+08:00, lastProcessTime=2023-05-15T14:00:44.830442+08:00 2023-05-15 14:00:54.859 INFO 11 --- [alarm-detect-12,,36b16acd2ef8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130434 ~ 1684130444 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130440=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:00:54.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:00:54.831938+08:00 2023-05-15 14:00:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,6223e68aba74] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:00:55.103 INFO 11 --- [pool-prometheus-second-schedule-2,,9d8365e70fb2] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:00:55.237 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,284a8cfb776d442c,99da1f21ad9d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=284a8cfb776d442c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:00:55.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,284a8cfb776d442c,99da1f21ad9d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=284a8cfb776d442c, duration=1 ms] 2023-05-15 14:00:55.899 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:55.905 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:55.907 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:55.909 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:56.641 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:56.642 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:00:56.885 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:57.274 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:00:57.531 INFO 11 --- [pool-ocp-schedules-17,,b3a370e83abf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:00:58.258 INFO 11 --- [pool-overview-container-7,,99d0cdfa4c55] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:00:58.505 INFO 11 --- [pool-scanner--15969501-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:00:59.356 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,13bcef9c5f7d4b8b,8e134c65fa12] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=13bcef9c5f7d4b8b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:00:59.362 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,13bcef9c5f7d4b8b,8e134c65fa12] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=13bcef9c5f7d4b8b, duration=6 ms] 2023-05-15 14:00:59.644 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:00:59.645 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:01:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:01:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130460 2023-05-15 14:01:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:01:00.050 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:01:00.304 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,92ae7332e69241b3,58161a9d67c6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=92ae7332e69241b3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:00.309 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,92ae7332e69241b3,58161a9d67c6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=92ae7332e69241b3, duration=6 ms] 2023-05-15 14:01:00.780 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 13:01:01.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.780 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=8, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2023-05-15 13:01:01.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.808 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect cluster hour metric, nodeCount=2, taskCount=3 2023-05-15 14:01:00.809 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.809 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect cluster hour metric, taskDefinitionId=8, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:00.823 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.823 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.827 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:01:00.827 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.827 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:00.832 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.832 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.842 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:01:00.842 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.842 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:00.847 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.847 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.855 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:01:00.855 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.855 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:31.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:00:31.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.862 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:01:00.862 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.862 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:00.869 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:01:00.0, currentDate=Mon May 15 14:01:00 CST 2023 2023-05-15 14:01:00.869 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:01:00.0} 2023-05-15 14:01:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:01:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:01.281 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131842, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.283 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131842, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:01:01.279+08:00, endTime=null} 2023-05-15 14:01:01.294 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131843, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.296 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131843, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:01:01.291+08:00, endTime=null} 2023-05-15 14:01:01.305 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfab8b9a804f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131842, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116247, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.279+08:00, sub_task_instance_id=131842}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.306 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131840, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.309 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131840, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:01:01.304+08:00, endTime=null} 2023-05-15 14:01:01.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,4940518fb0e8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131843, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116248, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.291+08:00, sub_task_instance_id=131843}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,4940518fb0e8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:01:01.319 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131841, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.321 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131841, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:01:01.316+08:00, endTime=null} 2023-05-15 14:01:01.326 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfab8b9a804f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:01:01.326 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfab8b9a804f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:01:01.328 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0e41f3f60ea3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131840, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116245, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.304+08:00, sub_task_instance_id=131840}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.330 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfab8b9a804f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:01:01.331 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131844, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.333 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131844, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:01:01.328+08:00, endTime=null} 2023-05-15 14:01:01.335 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfab8b9a804f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131842, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.336 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:01:01.337 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:01:01.337 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:01:01.337 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:01:01.337 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:01:01.339 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67f752487282] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131841, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116246, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.316+08:00, sub_task_instance_id=131841}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.341 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:01:01.341 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,0e41f3f60ea3] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131840, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.342 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:01:01.342 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:01:01.342 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:01:01.342 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:01:01.343 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:01:01.343 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:01:01.343 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:01:01.343 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:01:01.343 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:01:01.343 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:01:01.349 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131837, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:01.351 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131837, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-05-15T14:01:01.347+08:00, endTime=null} 2023-05-15 14:01:01.353 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3a33fc74b24c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131844, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116249, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.328+08:00, sub_task_instance_id=131844}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.354 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.356 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.357 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.358 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.359 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:01:01.360 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.360 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.360 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.360 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.361 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67f752487282] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:01:01.361 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67f752487282] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:01:01.361 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:01.361 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:01:01.361 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:01:01.361 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:01:01.362 INFO 11 --- [pool-3243289-thread-4,fb27d19334114ed7,7bd7328d26b5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:01:01.362 INFO 11 --- [pool-3243289-thread-2,fb27d19334114ed7,efed4061a160] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:01:01.362 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:01:01.362 INFO 11 --- [pool-3243289-thread-3,fb27d19334114ed7,13ce7dbd24a8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:01:01.362 INFO 11 --- [pool-3243289-thread-1,fb27d19334114ed7,1faf68c464d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:01:01.367 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.367 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.368 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:01:01.368 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:01:01.368 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67f752487282] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131841, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.368 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:01.368 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:01.368 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:01:01.369 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:01:01.369 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:01:01.371 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:01:01.372 INFO 11 --- [pool-3243289-thread-6,fb27d19334114ed7,16d739be96dc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:01:01.372 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:01:01.372 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:01:01.372 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3a33fc74b24c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131844, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.377 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:01:01.377 INFO 11 --- [pool-3243289-thread-5,fb27d19334114ed7,cefc7e38e838] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:01:01.378 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,4940518fb0e8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:01:01.381 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,163d1462c4a8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131837, context=Context{parallelIdx=-1, stringMap={begin_interval_time=2023-05-15T13:01:00.806+08:00, prohibit_rollback=false, task_instance_id=116244, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:01.346+08:00, end_interval_time=2023-05-15T14:01:00.806+08:00, sub_task_instance_id=131837}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:01:01.382 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,4940518fb0e8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131843, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.384 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,163d1462c4a8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131837, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,7b11d8d8eb10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130520, interval=60 2023-05-15 14:01:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,7898b293e9fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130520, interval=60 2023-05-15 14:01:01.525 INFO 11 --- [pool-prometheus-minute-schedule-2,,7898b293e9fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:01:01.525 INFO 11 --- [pool-prometheus-minute-schedule-2,,7898b293e9fb] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130461, count=6, elapsed=1682446331064 2023-05-15 14:01:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,7b11d8d8eb10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:01:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,7b11d8d8eb10] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130461, count=8, elapsed=1682446331064 2023-05-15 14:01:01.563 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116245, status:SUCCESS 2023-05-15 14:01:01.564 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116245, status:SUCCESS 2023-05-15 14:01:01.576 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116246, status:SUCCESS 2023-05-15 14:01:01.577 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116246, status:SUCCESS 2023-05-15 14:01:01.584 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116247, status:SUCCESS 2023-05-15 14:01:01.585 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116247, status:SUCCESS 2023-05-15 14:01:01.592 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116248, status:SUCCESS 2023-05-15 14:01:01.593 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116248, status:SUCCESS 2023-05-15 14:01:01.603 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116249, status:SUCCESS 2023-05-15 14:01:01.604 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116249, status:SUCCESS 2023-05-15 14:01:02.382 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131838, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:02.384 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131838, name=Generate cluster snapshot, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask, seriesId=1, startTime=2023-05-15T14:01:02.380+08:00, endTime=null} 2023-05-15 14:01:02.402 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131839, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:02.404 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131839, name=Generate cluster snapshot, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.diag.internal.task.GenerateSnapshotTask, seriesId=2, startTime=2023-05-15T14:01:02.400+08:00, endTime=null} 2023-05-15 14:01:02.411 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131838, context=Context{parallelIdx=0, stringMap={begin_interval_time=2023-05-15T13:01:00.806+08:00, prohibit_rollback=false, task_instance_id=116244, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:02.379+08:00, end_interval_time=2023-05-15T14:01:00.806+08:00, sub_task_instance_id=131838}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:01:02.429 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : start to generate snapshot for cluster: 1 2023-05-15 14:01:02.434 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131839, context=Context{parallelIdx=1, stringMap={begin_interval_time=2023-05-15T13:01:00.806+08:00, prohibit_rollback=false, task_instance_id=116244, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:02.400+08:00, end_interval_time=2023-05-15T14:01:00.806+08:00, sub_task_instance_id=131839}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:01:02.437 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : start to produce cluster snapshot data: cluster_id = 1 2023-05-15 14:01:02.437 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot, context=GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=1, name=ocpcluster, obClusterId=1, status=RUNNING}, snapshotId=167, beginIntervalTime=1684126860726000, beginIntervalTimeString=2023-05-15T13:01:00.726, endIntervalTime=1684130460806000, endIntervalTimeString=2023-05-15T14:01:00.806} 2023-05-15 14:01:02.437 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for sqlAuditSnapshotOperator 2023-05-15 14:01:02.454 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : start to generate snapshot for cluster: 2 2023-05-15 14:01:02.457 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : start to produce cluster snapshot data: cluster_id = 2 2023-05-15 14:01:02.457 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot, context=GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=2, name=obcluster, obClusterId=1, status=RUNNING}, snapshotId=145, beginIntervalTime=1684126860726000, beginIntervalTimeString=2023-05-15T13:01:00.726, endIntervalTime=1684130460806000, endIntervalTimeString=2023-05-15T14:01:00.806} 2023-05-15 14:01:02.458 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for sqlAuditSnapshotOperator 2023-05-15 14:01:02.469 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:02.469 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:02.469 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:02.470 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:02.472 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:02.472 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:02.473 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:02.474 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Produce base for 1, [Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.10, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), stopTime=0, startServiceTime=1683530134960216, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.10, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), stopTime=0, startServiceTime=1683530134960216, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.10, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), stopTime=0, startServiceTime=1683530134960216, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.10, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), stopTime=0, startServiceTime=1683530134960216, blockMigrateInTime=0, id=1))], between 1684126860726000:1684130460806000 2023-05-15 14:01:02.500 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:02.500 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:02.500 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:02.501 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:02.591 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Finished Producing base for cluster: 1, between 1684126860726000:1684130460806000 2023-05-15 14:01:02.591 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for sqlAuditSnapshotOperator, error count is 0 2023-05-15 14:01:02.591 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for CacheSnapshotOperator 2023-05-15 14:01:02.597 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:02.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:02.645 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:02.646 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:02.676 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:02.676 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:02.677 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for CacheSnapshotOperator, error count is 0 2023-05-15 14:01:02.677 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for LatchSnapshotOperator 2023-05-15 14:01:02.679 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:02.681 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:02.682 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Produce base for 2, [Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.11, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1681989951900812, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.12, svrPort=2982, innerPort=2981, withRootserver=false, status=ACTIVE, zone=zone2, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1683708604272665, blockMigrateInTime=0, id=7)), Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.13, svrPort=2982, innerPort=2981, withRootserver=false, status=ACTIVE, zone=zone3, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1681989952296456, blockMigrateInTime=0, id=3)), Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.11, svrPort=2982, innerPort=2981, withRootserver=true, status=ACTIVE, zone=zone1, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1681989951900812, blockMigrateInTime=0, id=1)), Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.12, svrPort=2982, innerPort=2981, withRootserver=false, status=ACTIVE, zone=zone2, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1683708604272665, blockMigrateInTime=0, id=7)), Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=ObServer(svrIp=10.xxx.xx.13, svrPort=2982, innerPort=2981, withRootserver=false, status=ACTIVE, zone=zone3, buildVersion=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), stopTime=0, startServiceTime=1681989952296456, blockMigrateInTime=0, id=3))], between 1684126860726000:1684130460806000 2023-05-15 14:01:02.685 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:02.800 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.o.SqlAuditStatOperator : Finished Producing base for cluster: 2, between 1684126860726000:1684130460806000 2023-05-15 14:01:02.800 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for sqlAuditSnapshotOperator, error count is 0 2023-05-15 14:01:02.800 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for CacheSnapshotOperator 2023-05-15 14:01:02.803 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:02.805 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:02.826 INFO 11 --- [pool-scanner--15969528-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.829 INFO 11 --- [pool-scanner--15969526-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.832 INFO 11 --- [pool-scanner--15969524-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.836 INFO 11 --- [pool-scanner--15969527-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.842 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for CacheSnapshotOperator, error count is 0 2023-05-15 14:01:02.842 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for LatchSnapshotOperator 2023-05-15 14:01:02.843 INFO 11 --- [pool-scanner--15969525-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.845 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:02.848 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:02.857 INFO 11 --- [pool-scanner--15969529-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:02.861 INFO 11 --- [pool-scanner--15969523-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1176 2023-05-15 14:01:04.540 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for LatchSnapshotOperator, error count is 0 2023-05-15 14:01:04.540 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ParameterSnapshotOperator 2023-05-15 14:01:04.545 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ParameterSnapshotOperator, error count is 0 2023-05-15 14:01:04.545 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RowLockSnapshotOperator 2023-05-15 14:01:04.548 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.552 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.566 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for LatchSnapshotOperator, error count is 0 2023-05-15 14:01:04.566 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ParameterSnapshotOperator 2023-05-15 14:01:04.569 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ParameterSnapshotOperator, error count is 0 2023-05-15 14:01:04.569 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RowLockSnapshotOperator 2023-05-15 14:01:04.572 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.574 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.595 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RowLockSnapshotOperator, error count is 0 2023-05-15 14:01:04.595 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ClusterInfoSnapshotOperator 2023-05-15 14:01:04.598 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RowLockSnapshotOperator, error count is 0 2023-05-15 14:01:04.599 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ClusterInfoSnapshotOperator 2023-05-15 14:01:04.601 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.o.ClusterInfoOperator : ob_hist_cluster_snapshot generate data success, clusterName = ocpcluster, snapshotId = 167, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.601 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ClusterInfoSnapshotOperator, error count is 0 2023-05-15 14:01:04.601 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for TenantInfoSnapshotOperator 2023-05-15 14:01:04.604 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.o.ClusterInfoOperator : ob_hist_cluster_snapshot generate data success, clusterName = obcluster, snapshotId = 145, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.604 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ClusterInfoSnapshotOperator, error count is 0 2023-05-15 14:01:04.604 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for TenantInfoSnapshotOperator 2023-05-15 14:01:04.614 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.615 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.616 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.623 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.624 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.624 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.624 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.625 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-05-15 14:01:04.629 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.629 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.630 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-05-15 14:01:04.649 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.operator.TenantInfoOperator : ob_hist_tenant snapshot generate data success, clusterName = obcluster, snapshotId = 145, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.649 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for TenantInfoSnapshotOperator, error count is 0 2023-05-15 14:01:04.649 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ReplicaInfoOperator 2023-05-15 14:01:04.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.operator.TenantInfoOperator : ob_hist_tenant snapshot generate data success, clusterName = ocpcluster, snapshotId = 167, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for TenantInfoSnapshotOperator, error count is 0 2023-05-15 14:01:04.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ReplicaInfoOperator 2023-05-15 14:01:04.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.664 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.664 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.666 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-05-15 14:01:04.670 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.671 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.671 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.672 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.672 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.677 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.677 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.678 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, TENANT_ID, MIN_CPU, MAX_CPU, MIN_MEMORY, MAX_MEMORY, MIN_IOPS, MAX_IOPS, IOPS_WEIGHT, DATA_DISK_IN_USE, LOG_DISK_IN_USE FROM (SELECT T1.UNIT_ID, T1.SVR_IP, T1.SVR_PORT, T1.ZONE, T2.TENANT_ID, T1.MIN_CPU, T1.MAX_CPU, T1.MIN_MEMORY, T1.MAX_MEMORY, T1.MIN_IOPS, T1.MAX_IOPS, T1.IOPS_WEIGHT, T1.DATA_DISK_IN_USE, T1.LOG_DISK_IN_USE FROM (SELECT UNIT_ID, SVR_IP, SVR_PORT, ZONE, SUM(MIN_CPU) AS MIN_CPU, SUM(MAX_CPU) AS MAX_CPU, SUM(MEMORY_SIZE) AS MIN_MEMORY, SUM(MEMORY_SIZE) AS MAX_MEMORY, SUM(MIN_IOPS) AS MIN_IOPS, SUM(MAX_IOPS) AS MAX_IOPS, SUM(IOPS_WEIGHT) AS IOPS_WEIGHT, SUM(DATA_DISK_IN_USE) AS DATA_DISK_IN_USE, SUM(LOG_DISK_IN_USE) AS LOG_DISK_IN_USE FROM oceanbase.GV$OB_UNITS GROUP BY UNIT_ID, SVR_IP, SVR_PORT ) T1 JOIN oceanbase.DBA_OB_UNITS T2 ON T1.UNIT_ID = T2.UNIT_ID AND T1.SVR_IP = T2.SVR_IP AND T1.SVR_PORT = T2.SVR_PORT) 2023-05-15 14:01:04.692 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ReplicaInfoOperator, error count is 0 2023-05-15 14:01:04.692 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SysStatSnapshotOperator 2023-05-15 14:01:04.696 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.699 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.707 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ReplicaInfoOperator, error count is 0 2023-05-15 14:01:04.707 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SysStatSnapshotOperator 2023-05-15 14:01:04.712 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.713 INFO 11 --- [pool-ocp-schedules-20,,1a29bc4b0f40] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:01:04.715 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.722 INFO 11 --- [pool-ocp-schedules-20,,1a29bc4b0f40] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:01:04.745 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.745 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:04.748 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.748 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:04.827 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:04.826985+08:00, lastProcessTime=2023-05-15T14:00:54.831938+08:00 2023-05-15 14:01:04.842 INFO 11 --- [alarm-detect-4,,6807151ba896] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130444 ~ 1684130454 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130450=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:04.858 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = obcluster, snapshotId = 145, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.858 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = obcluster, snapshotId = 145, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.858 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SysStatSnapshotOperator, error count is 0 2023-05-15 14:01:04.858 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ServerInfoSnapshotOperator 2023-05-15 14:01:04.860 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:04.826985+08:00 2023-05-15 14:01:04.942 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.943 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.944 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.944 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.945 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.956 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:04.956 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.962 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, DATA_DISK_CAPACITY AS DISK_CAPACITY, DATA_DISK_IN_USE AS DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2023-05-15 14:01:04.980 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = ocpcluster, snapshotId = 167, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.981 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.operator.SysStatOperator : ob_hist_sysstat snapshot generate data success, clusterName = ocpcluster, snapshotId = 167, endIntervalTime = 1684130460806000 2023-05-15 14:01:04.981 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SysStatSnapshotOperator, error count is 0 2023-05-15 14:01:04.981 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for ServerInfoSnapshotOperator 2023-05-15 14:01:04.993 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:04.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:04.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:04.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:04.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, DATA_DISK_CAPACITY AS DISK_CAPACITY, DATA_DISK_IN_USE AS DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.operator.ServerInfoOperator : ob_hist_server snapshot generate data success, clusterName = ocpcluster, snapshotId = 167, endIntervalTime = 1684130460806000 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ServerInfoSnapshotOperator, error count is 0 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for PartitionHitOperator 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.operator.ServerInfoOperator : ob_hist_server snapshot generate data success, clusterName = obcluster, snapshotId = 145, endIntervalTime = 1684130460806000 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for ServerInfoSnapshotOperator, error count is 0 2023-05-15 14:01:05.001 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for PartitionHitOperator 2023-05-15 14:01:05.005 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.006 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.007 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.010 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.049 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for PartitionHitOperator, error count is 0 2023-05-15 14:01:05.049 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RpcSnapshotOperator 2023-05-15 14:01:05.054 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for PartitionHitOperator, error count is 0 2023-05-15 14:01:05.054 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for RpcSnapshotOperator 2023-05-15 14:01:05.064 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:05.065 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:05.065 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:05.065 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:05.066 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:05.071 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:05.072 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:05.073 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:05.073 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:05.074 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:05.074 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:05.074 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:05.075 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id as ob_tenant_id, a.svr_ip as server_ip, a.pcode as packet_code, a.pcode_name as packet_name, a.count as outgoing_count, a.total_size as outgoing_size_byte, b.count as incoming_count, b.total_size as incoming_size_byte, a.failure as outgoing_fail_count, a.timeout as outgoing_timeout_count from gv$ob_rpc_outgoing a, gv$ob_rpc_incoming b where a.svr_ip=b.svr_ip and a.pcode=b.pcode and a.tenant_id=b.tenant_id and a.count > 0 and b.count>0 2023-05-15 14:01:05.079 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:05.079 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:05.080 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id as ob_tenant_id, a.svr_ip as server_ip, a.pcode as packet_code, a.pcode_name as packet_name, a.count as outgoing_count, a.total_size as outgoing_size_byte, b.count as incoming_count, b.total_size as incoming_size_byte, a.failure as outgoing_fail_count, a.timeout as outgoing_timeout_count from gv$ob_rpc_outgoing a, gv$ob_rpc_incoming b where a.svr_ip=b.svr_ip and a.pcode=b.pcode and a.tenant_id=b.tenant_id and a.count > 0 and b.count>0 2023-05-15 14:01:05.120 WARN 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.internal.operator.RpcOperator : Invalid timestamp for snapshot. last snapshot is ObHistSnapshotEntity(clusterId=1, obClusterId=1, clusterName=ocpcluster, snapshotId=167, version=0, beginIntervalTime=1684126860726000, endIntervalTime=1684130460806000, taskInstanceId=116244, flushElapsed=0, errorCount=-1, errorInfo=, createTime=null, updateTime=null), current snapshot context is GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=1, name=ocpcluster, obClusterId=1, status=RUNNING}, snapshotId=167, beginIntervalTime=1684126860726000, beginIntervalTimeString=2023-05-15T13:01:00.726, endIntervalTime=1684130460806000, endIntervalTimeString=2023-05-15T14:01:00.806} 2023-05-15 14:01:05.144 WARN 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.internal.operator.RpcOperator : Invalid timestamp for snapshot. last snapshot is ObHistSnapshotEntity(clusterId=2, obClusterId=1, clusterName=obcluster, snapshotId=145, version=0, beginIntervalTime=1684126860726000, endIntervalTime=1684130460806000, taskInstanceId=116244, flushElapsed=0, errorCount=-1, errorInfo=, createTime=null, updateTime=null), current snapshot context is GenerateSnapshotContext{clusterEntity=ObClusterEntity{id=2, name=obcluster, obClusterId=1, status=RUNNING}, snapshotId=145, beginIntervalTime=1684126860726000, beginIntervalTimeString=2023-05-15T13:01:00.726, endIntervalTime=1684130460806000, endIntervalTimeString=2023-05-15T14:01:00.806} 2023-05-15 14:01:05.393 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.396 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.439 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.442 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.451 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.454 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.462 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.465 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.487 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.488 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.490 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.491 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.506 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RpcSnapshotOperator, error count is 0 2023-05-15 14:01:05.506 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for MemoryOperator 2023-05-15 14:01:05.509 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.510 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.512 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.513 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.529 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.531 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.543 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for RpcSnapshotOperator, error count is 0 2023-05-15 14:01:05.544 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for MemoryOperator 2023-05-15 14:01:05.547 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.549 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.556 INFO 11 --- [pool-scanner--15969820-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=203 2023-05-15 14:01:05.561 INFO 11 --- [pool-scanner--15969819-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=203 2023-05-15 14:01:05.564 INFO 11 --- [pool-scanner--15969821-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=203 2023-05-15 14:01:05.646 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:05.647 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:05.748 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-05-15 14:01:05.779 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for MemoryOperator, error count is 0 2023-05-15 14:01:05.779 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SystemEventSnapshotOperator 2023-05-15 14:01:05.783 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.786 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.848 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for MemoryOperator, error count is 0 2023-05-15 14:01:05.848 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Start to generate snapshot data for SystemEventSnapshotOperator 2023-05-15 14:01:05.852 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:01:05.854 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:01:05.901 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:05.907 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:05.909 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:05.910 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:05.924 INFO 11 --- [pool-scanner--15969826-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1468 2023-05-15 14:01:05.943 INFO 11 --- [pool-scanner--15969827-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1468 2023-05-15 14:01:05.960 INFO 11 --- [pool-scanner--15969828-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=0, nodesFilledCount=1468 2023-05-15 14:01:06.101 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:05.0, nextRun=2023-05-15 14:01:05.0, currentDate=Mon May 15 14:01:06 CST 2023 2023-05-15 14:01:06.101 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:00:05.0, nextRunTime=2023-05-15 14:01:05.0} 2023-05-15 14:01:06.104 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:01:06.105 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:06.105 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:06.477 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131845, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:06.481 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131845, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:01:06.474+08:00, endTime=null} 2023-05-15 14:01:06.498 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,aba486254506] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131845, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116250, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:06.474+08:00, sub_task_instance_id=131845}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:06.520 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,2f0251370a10] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:01:06.521 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,a18cababbc2e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:01:06.522 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,4f0e2da5cbe2] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:01:06.566 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,4f0e2da5cbe2] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:01:06.578 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,2f0251370a10] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:01:06.583 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,a18cababbc2e] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:01:06.650 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,aba486254506] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131845, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:06.778 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116250, status:SUCCESS 2023-05-15 14:01:06.779 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116250, status:SUCCESS 2023-05-15 14:01:06.815 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.a.i.o.SystemEventOperator : [System event] store 57 targets 2023-05-15 14:01:06.834 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SystemEventSnapshotOperator, error count is 0 2023-05-15 14:01:06.834 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate all snapshot, failCount = 0, beans = [sqlAuditSnapshotOperator, CacheSnapshotOperator, LatchSnapshotOperator, ParameterSnapshotOperator, RowLockSnapshotOperator, ClusterInfoSnapshotOperator, TenantInfoSnapshotOperator, ReplicaInfoOperator, SysStatSnapshotOperator, ServerInfoSnapshotOperator, PartitionHitOperator, RpcSnapshotOperator, MemoryOperator, SystemEventSnapshotOperator], fail beans is None, fail info is Node 2023-05-15 14:01:06.847 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.a.i.o.SystemEventOperator : [System event] store 57 targets 2023-05-15 14:01:06.848 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7bbf85d61b9b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131838, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:06.856 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate snapshot data for SystemEventSnapshotOperator, error count is 0 2023-05-15 14:01:06.856 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.p.s.d.i.GenerateSnapshotService : Finish generate all snapshot, failCount = 0, beans = [sqlAuditSnapshotOperator, CacheSnapshotOperator, LatchSnapshotOperator, ParameterSnapshotOperator, RowLockSnapshotOperator, ClusterInfoSnapshotOperator, TenantInfoSnapshotOperator, ReplicaInfoOperator, SysStatSnapshotOperator, ServerInfoSnapshotOperator, PartitionHitOperator, RpcSnapshotOperator, MemoryOperator, SystemEventSnapshotOperator], fail beans is None, fail info is Node 2023-05-15 14:01:06.864 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,995dfafaec18] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131839, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:06.888 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:07.553 INFO 11 --- [pool-ocp-schedules-2,,019b720e8921] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:07.800 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116244, status:SUCCESS 2023-05-15 14:01:07.801 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116244, status:SUCCESS 2023-05-15 14:01:08.258 INFO 11 --- [pool-overview-container-1,,440a19921aae] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:08.507 INFO 11 --- [pool-scanner--15969832-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:01:08.647 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:08.648 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:09.944 INFO 11 --- [pool-ocp-schedules-9,,907ffe209902] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":77444} 2023-05-15 14:01:09.944 INFO 11 --- [pool-ocp-schedules-9,,907ffe209902] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:01:11.640 INFO 11 --- [pool-ocp-schedules-11,,b5d78ccdecf5] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:01:11.640 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:01:11.640 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:01:11.640 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:01:11.640 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:01:11.640 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:01:11.646 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:01:11.647 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:01:11.648 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:01:11.648 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:01:11.648 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:11.649 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:11.650 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:01:11.676 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:01:11.678 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:01:11.679 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:01:11.679 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:01:11.678431643+08:00 2023-05-15 14:01:11.679 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:01:11.679+08:00 2023-05-15 14:01:11.679 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:01:11.680 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:01:11.680135065+08:00 2023-05-15 14:01:11.681 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:01:11.680+08:00 2023-05-15 14:01:11.681 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:01:11.680736270+08:00 2023-05-15 14:01:11.681 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:01:11.681+08:00 2023-05-15 14:01:11.682 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:01:11.681784543+08:00 2023-05-15 14:01:11.682 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:01:11.682+08:00 2023-05-15 14:01:11.683 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:01:11.684 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:01:11.679+08:00 and status AVAILABLE 2023-05-15 14:01:11.684 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:01:11.681+08:00 and status AVAILABLE 2023-05-15 14:01:11.684 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:01:11.680+08:00 and status AVAILABLE 2023-05-15 14:01:11.685 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:01:11.685090370+08:00 2023-05-15 14:01:11.685 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:01:11.685+08:00 2023-05-15 14:01:11.687 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:01:11.682+08:00 and status AVAILABLE 2023-05-15 14:01:11.689 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:01:11.685+08:00 and status AVAILABLE 2023-05-15 14:01:11.690 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:01:11.690 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:01:11.690 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:01:11.711 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130471 0 0 , error= 2023-05-15 14:01:11.717 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130471 0 0 , error= 2023-05-15 14:01:11.720 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:01:11.715+08:00 2023-05-15 14:01:11.722 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130471 0 0 , error= 2023-05-15 14:01:11.722 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:01:11.723 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:01:11.722+08:00 2023-05-15 14:01:11.723 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:01:11.723+08:00 2023-05-15 14:01:11.723 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:01:11.748 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130471 0 0 , error= 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130471 0 0 , error= 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:01:11.754+08:00 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:01:11.754+08:00 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:01:11.715+08:00 and status AVAILABLE 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:01:11.722+08:00 and status AVAILABLE 2023-05-15 14:01:11.754 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:01:11.723+08:00 and status AVAILABLE 2023-05-15 14:01:11.757 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:01:11.758 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:01:11.758 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:01:11.759 INFO 11 --- [pool-ocp-schedules-6,,7a0ce84b3c41] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:11.759 INFO 11 --- [pool-ocp-schedules-6,,7a0ce84b3c41] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:11.760 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:01:11.754+08:00 and status AVAILABLE 2023-05-15 14:01:11.761 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:01:11.754+08:00 and status AVAILABLE 2023-05-15 14:01:11.762 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:01:11.763 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:01:11.770 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:01:11.770 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:01:11.771 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:01:11.772 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:01:11.775 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:01:11.797 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:01:11.798 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:01:11.798 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:01:11.800 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:01:11.804 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:01:11.846 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:01:11.851 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:01:11.852 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:01:11.860 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:01:11.863 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:01:11.864 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:01:11.865 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:01:11.875 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:01:11.877 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:01:11.879 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:01:11.882 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:01:11.883 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:01:11.886 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:01:11.894 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:01:11.902 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:01:11.926 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:01:11.932 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:01:11.935 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:01:11.941 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:01:11.946 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:01:11.949 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:01:11.952 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:01:11.961 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:01:11.964 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:01:11.965 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:01:11.967 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:01:11.977 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:01:11.985 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:01:11.992 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:01:12.011 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:01:12.012 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:01:12.022 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:01:12.024 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:01:12.024 INFO 11 --- [pool-Check host status1,,4773cc3a7758] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:01:12.037 INFO 11 --- [pool-Check host status14,,6ae1480eecda] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:01:12.039 INFO 11 --- [pool-Check host status8,,80f6d54119bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:01:12.047 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:01:12.060 INFO 11 --- [pool-Check host status13,,4fef1f3b4b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:01:12.092 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:01:12.105 INFO 11 --- [pool-Check host status30,,73d40b549f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:01:12.128 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:01:12.128 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:01:12.128 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:01:12.128 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:01:12.128 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:01:12.135 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:01:12.135 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:01:12.137 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:01:12.138 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:01:12.138 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:01:12.161 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:01:12.161 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:01:12.163 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:01:12.164 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:01:12.166 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:01:12.179 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:01:12.180 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:01:12.180 INFO 11 --- [pool-Check host status17,,f2c5ad2472d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:01:12.182 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:01:12.182 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:01:12.182 INFO 11 --- [pool-Check host status3,,1232eaa4e289] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:01:12.182 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:01:12.182 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:01:12.183 INFO 11 --- [pool-Check host status2,,6b8942836e04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:01:12.185 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:01:12.186 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:01:12.186 INFO 11 --- [pool-Check host status28,,62e4262365c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:01:12.187 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:01:12.187 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:01:12.187 INFO 11 --- [pool-Check host status19,,8d0492985f5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:01:12.276 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:14.269 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=54, affectRows=54 2023-05-15 14:01:14.269 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=54, validCount=54 2023-05-15 14:01:14.280 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=204, affectRows=204 2023-05-15 14:01:14.281 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=264, affectRows=264 2023-05-15 14:01:14.282 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=204, validCount=204 2023-05-15 14:01:14.283 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=264, validCount=264 2023-05-15 14:01:14.284 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=346, affectRows=346 2023-05-15 14:01:14.287 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=346, validCount=346 2023-05-15 14:01:14.300 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=1000, affectRows=1000 2023-05-15 14:01:14.308 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=1000, validCount=1000 2023-05-15 14:01:14.314 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130400, nodeCount=822, affectRows=822 2023-05-15 14:01:14.318 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130400, nodeCount=822, validCount=822 2023-05-15 14:01:14.622 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c6b4f2a9dbe74d65,799b5332c62c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=c6b4f2a9dbe74d65, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:14.628 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c6b4f2a9dbe74d65,799b5332c62c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=c6b4f2a9dbe74d65, duration=6 ms] 2023-05-15 14:01:14.649 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:14.650 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:14.782 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,f226f7cf8b77483f,e522ad1c0003] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=f226f7cf8b77483f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:14.788 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,f226f7cf8b77483f,e522ad1c0003] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=f226f7cf8b77483f, duration=6 ms] 2023-05-15 14:01:14.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:14.831560+08:00, lastProcessTime=2023-05-15T14:01:04.826985+08:00 2023-05-15 14:01:14.882 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:14.831560+08:00 2023-05-15 14:01:14.882 INFO 11 --- [alarm-detect-3,,89931bd07755] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130454 ~ 1684130464 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130460=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:14.978 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,befd0775da6b4361,2c238cb29e80] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=befd0775da6b4361, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:14.984 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,befd0775da6b4361,2c238cb29e80] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=befd0775da6b4361, duration=6 ms] 2023-05-15 14:01:15.904 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:15.909 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:15.910 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:15.912 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:16.891 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:17.575 INFO 11 --- [pool-ocp-schedules-12,,062276f41b8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:01:17.623 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.625 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:17.626 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:17.627 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.627 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:17.644 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:17.645 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:17.646 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:17.646 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.647 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:17.650 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:17.651 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:17.652 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:17.652 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.653 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:17.655 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:01:17.656 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:01:17.656 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:01:17.656 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:01:17.661 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:01:17.661 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.662 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:01:17.662 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.662 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:01:17.662 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:17.662 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:01:17.663 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:01:17.663 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:01:17.686 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T13:58:48, start=2023-05-15T13:55:47, end=2023-05-15T13:58:17, updates=103, granule=120000ms, delay=181000ms 2023-05-15 14:01:17.686 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T13:59:02, start=2023-05-15T13:55:47, end=2023-05-15T13:58:17, updates=106, granule=120000ms, delay=195000ms 2023-05-15 14:01:17.688 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:58:48, target=2023-05-15T13:58:17, granule=120000ms, delay=31000ms) 2023-05-15 14:01:17.688 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:58:17, target=2023-05-15T13:51:17, granule=600000ms, delay=420000ms) 2023-05-15 14:01:17.688 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:59:02, target=2023-05-15T13:58:17, granule=120000ms, delay=45000ms) 2023-05-15 14:01:17.688 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:58:17, target=2023-05-15T13:53:17, granule=600000ms, delay=300000ms) 2023-05-15 14:01:17.696 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T13:58:48, start=2023-05-15T13:55:47, end=2023-05-15T13:58:17, updates=128, granule=120000ms, delay=181000ms 2023-05-15 14:01:17.698 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T13:58:48, target=2023-05-15T13:58:17, granule=120000ms, delay=31000ms) 2023-05-15 14:01:17.699 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T13:58:17, target=2023-05-15T13:51:17, granule=600000ms, delay=420000ms) 2023-05-15 14:01:17.699 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=112ms, period=120000ms 2023-05-15 14:01:18.261 INFO 11 --- [pool-overview-container-6,,9e2fff1cdaee] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:18.509 INFO 11 --- [pool-scanner--15969932-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:20.608 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:21.0, nextRun=2023-05-15 14:01:20.0, currentDate=Mon May 15 14:01:20 CST 2023 2023-05-15 14:01:20.608 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:00:21.0, nextRunTime=2023-05-15 14:01:20.0} 2023-05-15 14:01:20.610 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:01:20.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:20.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:20.651 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:20.653 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:20.695 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131846, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:20.697 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131846, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:01:20.691+08:00, endTime=null} 2023-05-15 14:01:20.712 INFO 11 --- [pool-scanner--15969933-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:20.719 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b407af3e5720] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131846, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116251, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:20.691+08:00, sub_task_instance_id=131846}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:20.721 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b407af3e5720] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:01:20.721 INFO 11 --- [ForkJoinPool-66005-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:01:20.722 INFO 11 --- [ForkJoinPool-66005-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:01:20.730 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,b407af3e5720] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131846, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:20.737 INFO 11 --- [pool-scanner--15969934-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:20.760 INFO 11 --- [pool-scanner--15969935-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:20.782 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:20.789 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:20.830 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:20.834 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:20.852 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:20.855 INFO 11 --- [pool-ocp-schedules-12,,994a74dbaa5c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:21.096 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116251, status:SUCCESS 2023-05-15 14:01:21.098 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116251, status:SUCCESS 2023-05-15 14:01:21.793 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:21.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:21.797 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:23.653 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:23.655 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:24.830758+08:00, lastProcessTime=2023-05-15T14:01:14.831560+08:00 2023-05-15 14:01:24.873 INFO 11 --- [alarm-detect-3,,008885a1271d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130464 ~ 1684130474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130470=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:24.875 INFO 11 --- [alarm-detect-11,,737029b42b2d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130414 ~ 1684130474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388857.988, values={1684130460=388857.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:01:24.877 INFO 11 --- [alarm-detect-11,,737029b42b2d] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130414 ~ 1684130474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993655.358, values={1684130460=993655.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:24.878 INFO 11 --- [alarm-detect-9,,90068ac232c6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130414 ~ 1684130474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388857.938, values={1684130460=388857.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:01:24.878 INFO 11 --- [alarm-detect-9,,90068ac232c6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130414 ~ 1684130474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993655.107, values={1684130460=993655.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:24.907 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:24.830758+08:00 2023-05-15 14:01:25.779 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:26.0, nextRun=2023-05-15 14:01:25.0, currentDate=Mon May 15 14:01:25 CST 2023 2023-05-15 14:01:25.779 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:00:26.0, nextRunTime=2023-05-15 14:01:25.0} 2023-05-15 14:01:25.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:01:25.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:25.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:25.906 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:25.911 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:25.912 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:25.914 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:26.655 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:26.656 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:26.794 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131847, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:26.796 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131847, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:01:26.792+08:00, endTime=null} 2023-05-15 14:01:26.821 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131847, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116252, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:26.791+08:00, sub_task_instance_id=131847}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:01:26.870 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:01:26.871 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:01:26.872 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:01:26.872 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:01:26.873 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:01:26.878 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,fe7bf2c552b1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131847, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:26.894 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:27.215 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116252, status:SUCCESS 2023-05-15 14:01:27.216 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116252, status:SUCCESS 2023-05-15 14:01:27.279 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:27.598 INFO 11 --- [pool-ocp-schedules-2,,472b6054e436] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:28.259 INFO 11 --- [pool-overview-container-3,,6696254a5d8f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:28.510 INFO 11 --- [pool-scanner--15970036-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:29.270 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,fb94cb203634414a,858048f8c3f3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=fb94cb203634414a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:01:29.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,fb94cb203634414a,858048f8c3f3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=fb94cb203634414a, duration=3 ms] 2023-05-15 14:01:29.656 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:29.661 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:30.954 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:31.0, nextRun=2023-05-15 14:01:30.0, currentDate=Mon May 15 14:01:30 CST 2023 2023-05-15 14:01:30.954 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:00:31.0, nextRunTime=2023-05-15 14:01:30.0} 2023-05-15 14:01:30.962 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:01:30.962 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:30.962 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:30.962 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:30.977 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:01.0, nextRun=2023-05-15 14:01:30.0, currentDate=Mon May 15 14:01:30 CST 2023 2023-05-15 14:01:30.977 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:01:01.0, nextRunTime=2023-05-15 14:01:30.0} 2023-05-15 14:01:30.982 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:01:30.982 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:01:30.982 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:01:31.767 INFO 11 --- [pool-ocp-schedules-1,,ac81977e5ce8] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:31.768 INFO 11 --- [pool-ocp-schedules-1,,ac81977e5ce8] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:31.884 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131850, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:31.886 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131850, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:01:31.881+08:00, endTime=null} 2023-05-15 14:01:31.902 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131848, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:31.904 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131848, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:01:31.899+08:00, endTime=null} 2023-05-15 14:01:31.909 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,91db02bf3f39] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131850, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116254, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:31.881+08:00, sub_task_instance_id=131850}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:01:31.909 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,91db02bf3f39] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:01:31.924 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131849, operation:EXECUTE, state: RUNNING 2023-05-15 14:01:31.927 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131849, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:01:31.919+08:00, endTime=null} 2023-05-15 14:01:31.938 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131848, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116253, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:31.899+08:00, sub_task_instance_id=131848}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:01:31.942 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:01:31.942 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:01:31.942 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:01:31.942 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:01:31.943 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:01:31.946 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:01:31.958 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131849, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116253, task_operation=execute, latest_execution_start_time=2023-05-15T14:01:31.918+08:00, sub_task_instance_id=131849}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:01:31.962 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.962 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.963 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.963 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.964 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.964 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.964 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.964 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.965 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.965 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.968 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.969 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.970 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:01:31.971 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.971 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.971 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.971 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:01:31.971 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:01:31.972 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:01:31.972 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.972 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.972 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:01:31.973 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:01:31.973 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:01:31.974 INFO 11 --- [pool-3243473-thread-3,fb27d19334114ed7,cd70e9b812dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:01:31.974 INFO 11 --- [pool-3243473-thread-1,fb27d19334114ed7,744d9369763b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-2,fb27d19334114ed7,06f43760120b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-6,fb27d19334114ed7,5b317ca76f1c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:01:31.976 INFO 11 --- [pool-3243473-thread-5,fb27d19334114ed7,11dcd304d554] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:01:31.980 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.980 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.980 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.980 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.981 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.981 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.982 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:01:31.982 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.982 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:31.983 INFO 11 --- [pool-ocp-schedules-3,,a80ec2033e54] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:01:31.984 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.984 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.984 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:01:31.984 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:01:31.985 INFO 11 --- [pool-ocp-schedules-3,,a80ec2033e54] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:01:31.985 INFO 11 --- [pool-3243473-thread-4,fb27d19334114ed7,5c139064e328] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:01:31.986 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,91db02bf3f39] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:01:31.992 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,91db02bf3f39] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131850, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:31.993 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:31.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:31.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.001 INFO 11 --- [pool-ocp-schedules-10,,5de1cc1d7af4] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:01:32.002 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.003 INFO 11 --- [pool-ocp-schedules-10,,5de1cc1d7af4] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:01:32.003 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.003 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.004 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.005 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.005 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.006 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.006 INFO 11 --- [pool-ocp-schedules-12,,b1fde98ac0a9] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:01:32.007 INFO 11 --- [ForkJoinPool-66006-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:01:32.009 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.009 INFO 11 --- [pool-ocp-schedules-12,,b1fde98ac0a9] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:01:32.009 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.009 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.010 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.010 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.010 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.010 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.014 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.014 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.015 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.019 INFO 11 --- [ForkJoinPool-66006-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:01:32.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.026 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.030 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.030 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.031 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:01:32.033 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.034 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.034 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.034 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.035 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.035 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.035 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.035 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.036 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:01:32.040 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.049 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.050 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.050 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.051 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.052 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.052 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.053 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.053 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.054 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.054 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:01:32.055 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.055 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.055 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:01:32.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:01:32.070 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.070 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.071 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.073 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.073 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.073 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.073 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.074 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.075 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.075 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.075 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:01:32.076 WARN 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:01:32.077 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.077 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.078 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:32.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.079 INFO 11 --- [ForkJoinPool-66008-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:01:32.079 INFO 11 --- [ForkJoinPool-66008-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:01:32.079 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:01:32.080 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:01:32.085 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.086 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.086 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.089 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.089 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:01:32.091 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.092 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.092 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:01:32.099 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.103 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:01:32.104 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:01:32.110 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.110 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.111 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.111 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.111 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.112 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.112 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.112 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.120 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.122 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:01:32.122 INFO 11 --- [ForkJoinPool-66008-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:01:32.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.124 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.124 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.124 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.124 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.125 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.125 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.125 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.125 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.130 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.131 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.131 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.131 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.132 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.132 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.132 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.132 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:01:32.135 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.135 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.135 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:01:32.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.137 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:01:32.138 INFO 11 --- [ForkJoinPool-66008-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:01:32.139 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:01:32.151 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.152 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:01:32.152 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.155 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.155 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.156 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.156 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.156 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.156 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.156 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.157 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.157 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.157 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.157 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.157 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.162 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.165 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.165 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:01:32.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.169 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.169 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.169 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.170 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.170 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.170 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.179 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.179 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:01:32.186 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.188 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.188 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.189 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.190 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.191 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.191 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.192 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.192 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.192 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.192 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:01:32.196 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.196 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.197 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.197 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.198 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.198 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.198 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.198 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.199 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.199 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.199 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.203 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.203 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:01:32.210 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.210 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.211 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.211 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.212 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.213 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.214 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.214 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.214 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.214 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:01:32.214 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.215 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.215 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.215 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:01:32.219 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.219 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.220 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.220 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:01:32.223 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:01:32.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.226 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.226 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.226 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.226 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.226 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.227 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.227 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.227 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.227 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.227 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.228 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:01:32.228 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.228 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:01:32.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:01:32.232 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.232 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.233 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.250 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:01:32.250 INFO 11 --- [ForkJoinPool-66008-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:01:32.253 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:01:32.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.253 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.254 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.254 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.255 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:01:32.261 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.261 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.262 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.266 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:01:32.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.269 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:01:32.276 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.278 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:01:32.279 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:01:32.279 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.279 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:01:32.280 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:01:32.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:01:32.294 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.296 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:01:32.296 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:01:32.296 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:01:32.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:01:32.297 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:01:32.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:01:32.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:01:32.304 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2cf5c16aeecb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131849, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:32.308 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:01:32.311 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:01:32.311 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:01:32.311 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:01:32.316 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4ae6dfa7359d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131848, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:01:32.326 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116253, status:SUCCESS 2023-05-15 14:01:32.328 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116253, status:SUCCESS 2023-05-15 14:01:32.339 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116254, status:SUCCESS 2023-05-15 14:01:32.340 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116254, status:SUCCESS 2023-05-15 14:01:32.661 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:32.663 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:32.845 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,79c41b507b074443,20dcc77c74c0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=79c41b507b074443, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:32.850 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,79c41b507b074443,20dcc77c74c0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=79c41b507b074443, duration=6 ms] 2023-05-15 14:01:34.202 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,f0de9b7312e941da,398ebe1ed0f2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=f0de9b7312e941da, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:34.207 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,f0de9b7312e941da,398ebe1ed0f2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=f0de9b7312e941da, duration=6 ms] 2023-05-15 14:01:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:34.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:34.831264+08:00, lastProcessTime=2023-05-15T14:01:24.830758+08:00 2023-05-15 14:01:34.871 INFO 11 --- [alarm-detect-1,,16139bf1367d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130474 ~ 1684130484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.723, values={1684130480=245.723}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:34.911 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:34.831264+08:00 2023-05-15 14:01:35.663 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:35.664 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:35.908 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:35.912 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:35.915 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:35.917 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:36.897 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:37.621 INFO 11 --- [pool-ocp-schedules-19,,67ac9c13b8f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:38.259 INFO 11 --- [pool-overview-container-4,,ae14be18a9e0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:38.507 INFO 11 --- [pool-scanner--15970144-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:38.664 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:38.665 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:41.665 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:41.667 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:42.281 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:44.667 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:44.668 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:44.831764+08:00, lastProcessTime=2023-05-15T14:01:34.831264+08:00 2023-05-15 14:01:44.878 INFO 11 --- [alarm-detect-3,,d2916e2f0447] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130484 ~ 1684130494 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130490=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:44.912 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:44.831764+08:00 2023-05-15 14:01:45.472 INFO 11 --- [pool-ocp-schedules-9,,a3b2ea814d07] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:01:45.911 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:45.915 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:45.918 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:45.920 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:46.900 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:47.642 INFO 11 --- [pool-ocp-schedules-1,,8ca7f17ada8f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:47.668 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:47.669 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:48.261 INFO 11 --- [pool-overview-container-8,,f62c8aae01b2] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:48.509 INFO 11 --- [pool-scanner--15970244-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:48.521 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,3959ec2670194324,0acff380dea4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=3959ec2670194324, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:48.526 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,3959ec2670194324,0acff380dea4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=3959ec2670194324, duration=6 ms] 2023-05-15 14:01:50.670 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:50.671 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:51.772 INFO 11 --- [pool-ocp-schedules-6,,ebf9005eb45d] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:51.772 INFO 11 --- [pool-ocp-schedules-6,,ebf9005eb45d] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:01:51.795 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:51.797 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:51.799 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,3ad6496234ec] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":80134} 2023-05-15 14:01:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,3ad6496234ec] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:01:53.671 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:53.672 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:01:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:01:54.831140+08:00, lastProcessTime=2023-05-15T14:01:44.831764+08:00 2023-05-15 14:01:54.864 INFO 11 --- [alarm-detect-14,,bfa89c598244] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130494 ~ 1684130504 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130500=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:01:54.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:01:54.831140+08:00 2023-05-15 14:01:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,e48ec1d90189] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:01:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,c901cbc13dae] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:01:55.915 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:55.921 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:55.923 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:55.925 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:56.079 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,4ed391c23e414586,a41f00fa7145] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=4ed391c23e414586, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:01:56.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,4ed391c23e414586,a41f00fa7145] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=4ed391c23e414586, duration=7 ms] 2023-05-15 14:01:56.673 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:56.674 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:01:56.904 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:57.284 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:01:57.664 INFO 11 --- [pool-ocp-schedules-17,,4054f8d69d7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:01:58.260 INFO 11 --- [pool-overview-container-6,,9a8e7d40dd3f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:01:58.510 INFO 11 --- [pool-scanner--15970338-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:01:59.674 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:01:59.675 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:02:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:02:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:02:00.001 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130520 2023-05-15 14:02:00.040 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:02:00.848 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.848 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.856 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:02:00.856 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.856 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.856 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:00.868 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:01.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.868 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:01:01.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:02:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.872 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:00.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:01.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:01:01.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.892 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:02:00.892 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.892 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:00.897 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:01.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.897 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:01:01.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.911 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:02:00.911 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.911 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:00.917 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:31.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:01:31.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:02:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.921 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:00.929 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:01.0, nextRun=2023-05-15 14:02:00.0, currentDate=Mon May 15 14:02:00 CST 2023 2023-05-15 14:02:00.929 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:01:01.0, nextRunTime=2023-05-15 14:02:00.0} 2023-05-15 14:02:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:02:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:01.352 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131856, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.354 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131856, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:02:01.350+08:00, endTime=null} 2023-05-15 14:02:01.367 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131857, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.370 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131857, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:02:01.364+08:00, endTime=null} 2023-05-15 14:02:01.375 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e4136ce29b54] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131856, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116259, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.350+08:00, sub_task_instance_id=131856}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.375 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e4136ce29b54] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:02:01.388 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131851, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.390 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131851, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:02:01.385+08:00, endTime=null} 2023-05-15 14:02:01.394 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,88fd9061a514] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131857, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116260, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.364+08:00, sub_task_instance_id=131857}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:02:01.402 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:02:01.403 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131854, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.406 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131854, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:02:01.399+08:00, endTime=null} 2023-05-15 14:02:01.413 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,88fd9061a514] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131857, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.414 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f439f53cea57] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131851, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116255, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.384+08:00, sub_task_instance_id=131851}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.414 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.415 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.416 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.417 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131855, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:02:01.417 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:02:01.418 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:02:01.418 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:01.418 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.419 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:02:01.419 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131855, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:02:01.415+08:00, endTime=null} 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-3,fb27d19334114ed7,21c1f6eadc07] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-4,fb27d19334114ed7,ac5034156d39] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:02:01.420 INFO 11 --- [pool-3243640-thread-2,fb27d19334114ed7,e9bc937a4c8e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:02:01.421 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:02:01.421 INFO 11 --- [pool-3243640-thread-1,fb27d19334114ed7,106476ab4e6d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:02:01.421 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.421 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.421 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.422 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.422 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.423 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.423 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.424 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:01.424 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22599d0d1f2a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131854, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116257, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.399+08:00, sub_task_instance_id=131854}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.429 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.429 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.429 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131852, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.429 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:01.429 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:01.429 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:02:01.430 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:02:01.431 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131852, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:02:01.427+08:00, endTime=null} 2023-05-15 14:02:01.431 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f439f53cea57] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131851, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.434 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d66b2f078491] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131855, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116258, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.415+08:00, sub_task_instance_id=131855}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.434 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:02:01.434 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:02:01.434 INFO 11 --- [pool-3243640-thread-6,fb27d19334114ed7,755c7a7d5c73] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:02:01.434 INFO 11 --- [pool-3243640-thread-5,fb27d19334114ed7,da28bce53086] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:02:01.435 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e4136ce29b54] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:02:01.439 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e4136ce29b54] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131856, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.439 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131853, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:01.440 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131853, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:02:01.437+08:00, endTime=null} 2023-05-15 14:02:01.441 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22599d0d1f2a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:02:01.441 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22599d0d1f2a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:02:01.445 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22599d0d1f2a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131854, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.446 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,83bd3c8b56ba] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131852, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116255, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.427+08:00, sub_task_instance_id=131852}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.450 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d66b2f078491] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:02:01.450 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d66b2f078491] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:02:01.452 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d66b2f078491] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:02:01.457 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,d66b2f078491] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131855, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.457 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2a65e76e2e5a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131853, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116256, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:01.437+08:00, sub_task_instance_id=131853}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:01.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,83bd3c8b56ba] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131852, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.463 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:02:01.463 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:02:01.463 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:02:01.464 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:02:01.464 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:02:01.468 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2a65e76e2e5a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131853, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:01.469 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:02:01.469 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:02:01.469 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:02:01.469 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:02:01.470 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:02:01.492 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:02:01.497 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:02:01.497 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:02:01.498 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:02:01.502 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:02:01.502 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:02:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,104466c3e427] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130580, interval=60 2023-05-15 14:02:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,3a12d394916a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130580, interval=60 2023-05-15 14:02:01.514 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:02:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,104466c3e427] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:02:01.525 INFO 11 --- [pool-prometheus-minute-schedule-2,,3a12d394916a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:02:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,104466c3e427] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130521, count=8, elapsed=1682446391004 2023-05-15 14:02:01.525 INFO 11 --- [pool-prometheus-minute-schedule-2,,3a12d394916a] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130521, count=6, elapsed=1682446391004 2023-05-15 14:02:01.960 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116255, status:SUCCESS 2023-05-15 14:02:01.974 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116255, status:SUCCESS 2023-05-15 14:02:02.030 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116256, status:SUCCESS 2023-05-15 14:02:02.032 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116256, status:SUCCESS 2023-05-15 14:02:02.056 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116257, status:SUCCESS 2023-05-15 14:02:02.057 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116257, status:SUCCESS 2023-05-15 14:02:02.087 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116258, status:SUCCESS 2023-05-15 14:02:02.089 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116258, status:SUCCESS 2023-05-15 14:02:02.098 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116259, status:SUCCESS 2023-05-15 14:02:02.099 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116259, status:SUCCESS 2023-05-15 14:02:02.113 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116260, status:SUCCESS 2023-05-15 14:02:02.115 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116260, status:SUCCESS 2023-05-15 14:02:02.157 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,071e117912b5407d,5d4397578a32] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=071e117912b5407d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:02.165 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,071e117912b5407d,5d4397578a32] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=071e117912b5407d, duration=8 ms] 2023-05-15 14:02:02.675 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:02.676 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:03.301 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,680e0c255b3c4354,ff37ceda003b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=680e0c255b3c4354, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:02:03.306 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,680e0c255b3c4354,ff37ceda003b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=680e0c255b3c4354, duration=5 ms] 2023-05-15 14:02:04.713 INFO 11 --- [pool-ocp-schedules-13,,9ecf7589bf43] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:02:04.719 INFO 11 --- [pool-ocp-schedules-13,,9ecf7589bf43] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:02:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:04.830344+08:00, lastProcessTime=2023-05-15T14:01:54.831140+08:00 2023-05-15 14:02:04.851 INFO 11 --- [alarm-detect-6,,d3c8f51bfcbf] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130504 ~ 1684130514 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130510=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:04.878 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:04.830344+08:00 2023-05-15 14:02:05.138 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:06.0, nextRun=2023-05-15 14:02:05.0, currentDate=Mon May 15 14:02:05 CST 2023 2023-05-15 14:02:05.138 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:01:06.0, nextRunTime=2023-05-15 14:02:05.0} 2023-05-15 14:02:05.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:02:05.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:05.143 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:05.503 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131858, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:05.504 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131858, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:02:05.500+08:00, endTime=null} 2023-05-15 14:02:05.523 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ea284ccb0c5d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131858, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116261, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:05.500+08:00, sub_task_instance_id=131858}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:05.544 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,cdd0aff0dae1] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:02:05.544 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,3f1ae48a4677] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:02:05.546 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,1ec006d0dbc5] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:02:05.584 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,3f1ae48a4677] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:02:05.589 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,cdd0aff0dae1] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:02:05.589 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,1ec006d0dbc5] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:02:05.599 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ea284ccb0c5d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131858, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:05.676 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:05.677 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:05.921 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:05.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:05.926 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:05.928 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:06.190 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116261, status:SUCCESS 2023-05-15 14:02:06.191 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116261, status:SUCCESS 2023-05-15 14:02:06.907 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:07.682 INFO 11 --- [pool-ocp-schedules-9,,41ed3d67016c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:08.099 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e2e04a652c9b48fa,aeb95e8a0f86] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=e2e04a652c9b48fa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:08.104 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e2e04a652c9b48fa,aeb95e8a0f86] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=e2e04a652c9b48fa, duration=6 ms] 2023-05-15 14:02:08.260 INFO 11 --- [pool-overview-container-2,,19f9f319b283] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:02:08.260 INFO 11 --- [pool-overview-container-2,,19f9f319b283] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:08.511 INFO 11 --- [pool-scanner--15970431-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:02:08.678 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:08.679 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:10.111 INFO 11 --- [pool-ocp-schedules-5,,4a3d785d0804] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:02:11.679 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:11.681 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:11.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,0ac9104867164786,15d050300256] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=0ac9104867164786, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:11.708 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,0ac9104867164786,15d050300256] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=0ac9104867164786, duration=6 ms] 2023-05-15 14:02:11.778 INFO 11 --- [pool-ocp-schedules-20,,23b50ca3d7b3] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:11.778 INFO 11 --- [pool-ocp-schedules-20,,23b50ca3d7b3] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:12.203 INFO 11 --- [pool-ocp-schedules-3,,5c5fa0f3fea3] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:02:12.203 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:02:12.203 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:02:12.203 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:02:12.203 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:02:12.203 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:02:12.210 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:02:12.210 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:02:12.211 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:02:12.211 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:02:12.211 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:02:12.235 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:02:12.236 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:02:12.237 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:02:12.237 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:02:12.237 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:02:12.237 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:02:12.237378589+08:00 2023-05-15 14:02:12.238 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:02:12.237+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:02:12.238633743+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:02:12.239+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:02:12.239069015+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:02:12.239+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:02:12.239286099+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:02:12.239+08:00 2023-05-15 14:02:12.239 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:02:12.239405925+08:00 2023-05-15 14:02:12.240 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:02:12.239+08:00 2023-05-15 14:02:12.241 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:02:12.237+08:00 and status AVAILABLE 2023-05-15 14:02:12.242 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:02:12.239+08:00 and status AVAILABLE 2023-05-15 14:02:12.242 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:02:12.239+08:00 and status AVAILABLE 2023-05-15 14:02:12.242 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:02:12.239+08:00 and status AVAILABLE 2023-05-15 14:02:12.242 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:02:12.239+08:00 and status AVAILABLE 2023-05-15 14:02:12.247 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:02:12.247 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:02:12.247 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:02:12.248 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:02:12.248 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:02:12.259 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130532 0 0 , error= 2023-05-15 14:02:12.259 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130532 0 0 , error= 2023-05-15 14:02:12.259 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130532 0 0 , error= 2023-05-15 14:02:12.272 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130532 0 0 , error= 2023-05-15 14:02:12.274 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130532 0 0 , error= 2023-05-15 14:02:12.275 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:02:12.268+08:00 2023-05-15 14:02:12.275 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:02:12.268+08:00 2023-05-15 14:02:12.275 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:02:12.268+08:00 2023-05-15 14:02:12.275 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:02:12.275+08:00 2023-05-15 14:02:12.275 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:02:12.275+08:00 2023-05-15 14:02:12.278 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:02:12.275+08:00 and status AVAILABLE 2023-05-15 14:02:12.278 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:02:12.275+08:00 and status AVAILABLE 2023-05-15 14:02:12.279 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:02:12.268+08:00 and status AVAILABLE 2023-05-15 14:02:12.279 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:02:12.268+08:00 and status AVAILABLE 2023-05-15 14:02:12.279 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:02:12.268+08:00 and status AVAILABLE 2023-05-15 14:02:12.281 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:02:12.287 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:12.292 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:02:12.319 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:02:12.321 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:02:12.321 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:02:12.322 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:02:12.322 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:02:12.334 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:02:12.334 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:02:12.334 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:02:12.335 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:02:12.359 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:02:12.360 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:02:12.362 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:02:12.362 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:02:12.375 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:02:12.385 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:02:12.399 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:02:12.406 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:02:12.410 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:02:12.412 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:02:12.417 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:02:12.420 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:02:12.430 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:02:12.431 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:02:12.439 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:02:12.446 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:02:12.451 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:02:12.458 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:02:12.464 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:02:12.472 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:02:12.477 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:02:12.488 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:02:12.492 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:02:12.496 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:02:12.498 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:02:12.502 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:02:12.504 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:02:12.511 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:02:12.514 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:02:12.534 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:02:12.534 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:02:12.556 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:02:12.557 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:02:12.573 INFO 11 --- [pool-Check host status22,,0e5162238015] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:02:12.573 INFO 11 --- [pool-Check host status26,,ac73f4db36db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:02:12.579 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:02:12.589 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:02:12.593 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:02:12.598 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:02:12.605 INFO 11 --- [pool-Check host status10,,d2bf673be46b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:02:12.611 INFO 11 --- [pool-Check host status9,,3d39d7309682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:02:12.654 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:02:12.738 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:02:12.754 INFO 11 --- [pool-Check host status12,,d3f1597c428e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:02:12.773 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:02:12.773 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:02:12.773 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:02:12.773 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:02:12.773 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:02:12.779 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:02:12.780 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:02:12.780 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:02:12.780 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:02:12.781 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:02:12.808 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:02:12.808 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:02:12.811 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:02:12.812 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:02:12.812 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status16,,db8c8ebaed44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:02:12.829 INFO 11 --- [pool-Check host status18,,243c6b71e4ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:02:12.830 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:02:12.830 INFO 11 --- [pool-Check host status15,,06a189c986ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:02:12.832 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:02:12.832 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:02:12.832 INFO 11 --- [pool-Check host status24,,f8dc748cb1da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:02:12.833 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:02:12.833 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:02:12.833 INFO 11 --- [pool-Check host status25,,8c99fde795e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:02:14.313 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130460, nodeCount=690, affectRows=690 2023-05-15 14:02:14.314 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130460, nodeCount=1000, affectRows=1000 2023-05-15 14:02:14.315 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130460, nodeCount=1000, affectRows=1000 2023-05-15 14:02:14.320 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130460, nodeCount=690, validCount=690 2023-05-15 14:02:14.322 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130460, nodeCount=1000, validCount=1000 2023-05-15 14:02:14.323 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130460, nodeCount=1000, validCount=1000 2023-05-15 14:02:14.681 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:14.682 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:14.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:14.831418+08:00, lastProcessTime=2023-05-15T14:02:04.830344+08:00 2023-05-15 14:02:14.867 INFO 11 --- [alarm-detect-12,,20bc040cca00] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130514 ~ 1684130524 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130520=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:14.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:14.831418+08:00 2023-05-15 14:02:15.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:15.929 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:15.936 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:15.942 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:16.910 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:17.683 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:17.684 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:17.709 INFO 11 --- [pool-ocp-schedules-16,,f1ee4608442f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:18.260 INFO 11 --- [pool-overview-container-7,,204fe8942052] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:18.511 INFO 11 --- [pool-scanner--15970531-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:20.217 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,3e09f12c58b74664,79baec274f7a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=3e09f12c58b74664, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:20.223 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,3e09f12c58b74664,79baec274f7a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=3e09f12c58b74664, duration=6 ms] 2023-05-15 14:02:20.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:21.0, nextRun=2023-05-15 14:02:20.0, currentDate=Mon May 15 14:02:20 CST 2023 2023-05-15 14:02:20.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:01:21.0, nextRunTime=2023-05-15 14:02:20.0} 2023-05-15 14:02:20.605 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:02:20.605 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:20.605 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:20.684 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:20.685 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:20.736 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131859, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:20.738 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131859, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:02:20.734+08:00, endTime=null} 2023-05-15 14:02:20.760 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,691cf0d00ea6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131859, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116262, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:20.734+08:00, sub_task_instance_id=131859}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:20.763 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,691cf0d00ea6] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:02:20.763 INFO 11 --- [ForkJoinPool-66011-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:02:20.764 INFO 11 --- [ForkJoinPool-66011-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:02:20.769 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,691cf0d00ea6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131859, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:20.895 INFO 11 --- [pool-scanner--15970532-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:20.918 INFO 11 --- [pool-scanner--15970533-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:20.938 INFO 11 --- [pool-scanner--15970534-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:20.960 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:20.966 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:21.016 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:21.021 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:21.044 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:21.048 INFO 11 --- [pool-ocp-schedules-9,,beebc9f836e4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:21.523 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116262, status:SUCCESS 2023-05-15 14:02:21.524 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116262, status:SUCCESS 2023-05-15 14:02:21.797 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:21.800 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:21.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:22.417 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5a6d6b23b56344c2,f54ac8d1416c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=5a6d6b23b56344c2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:22.424 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5a6d6b23b56344c2,f54ac8d1416c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=5a6d6b23b56344c2, duration=7 ms] 2023-05-15 14:02:23.686 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:23.687 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:24.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:24.827669+08:00, lastProcessTime=2023-05-15T14:02:14.831418+08:00 2023-05-15 14:02:24.865 INFO 11 --- [alarm-detect-9,,93ec59ac6a60] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130524 ~ 1684130534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130530=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:24.866 INFO 11 --- [alarm-detect-6,,b33a168570aa] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130474 ~ 1684130534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388917.988, values={1684130520=388917.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:02:24.867 INFO 11 --- [alarm-detect-6,,b33a168570aa] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130474 ~ 1684130534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993715.358, values={1684130520=993715.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:24.869 INFO 11 --- [alarm-detect-7,,e52e18ce0617] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130474 ~ 1684130534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388917.938, values={1684130520=388917.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:02:24.871 INFO 11 --- [alarm-detect-7,,e52e18ce0617] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130474 ~ 1684130534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993715.107, values={1684130520=993715.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:24.905 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:24.827669+08:00 2023-05-15 14:02:25.759 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:26.0, nextRun=2023-05-15 14:02:25.0, currentDate=Mon May 15 14:02:25 CST 2023 2023-05-15 14:02:25.759 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:01:26.0, nextRunTime=2023-05-15 14:02:25.0} 2023-05-15 14:02:25.770 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:02:25.770 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:25.770 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:25.826 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131860, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:25.828 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131860, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:02:25.824+08:00, endTime=null} 2023-05-15 14:02:25.850 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131860, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116263, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:25.824+08:00, sub_task_instance_id=131860}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:02:25.888 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:02:25.889 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:02:25.890 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:02:25.890 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:02:25.891 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:02:25.895 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b82b9c3c4bce] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131860, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:25.929 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:25.936 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:25.942 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:25.948 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:26.628 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116263, status:SUCCESS 2023-05-15 14:02:26.629 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116263, status:SUCCESS 2023-05-15 14:02:26.687 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:26.688 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:26.914 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:27.321 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:27.733 INFO 11 --- [pool-ocp-schedules-8,,375689522d75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:28.260 INFO 11 --- [pool-overview-container-1,,bcc6e4c9f56c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:28.511 INFO 11 --- [pool-scanner--15970635-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:29.689 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:29.689 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:30.925 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:01:31.0, nextRun=2023-05-15 14:02:30.0, currentDate=Mon May 15 14:02:30 CST 2023 2023-05-15 14:02:30.926 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:01:31.0, nextRunTime=2023-05-15 14:02:30.0} 2023-05-15 14:02:30.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:02:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:30.934 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:30.945 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:02:30.0, currentDate=Mon May 15 14:02:30 CST 2023 2023-05-15 14:02:30.945 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:02:30.0} 2023-05-15 14:02:30.948 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:02:30.948 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:02:30.948 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:02:31.783 INFO 11 --- [pool-ocp-schedules-9,,1eb9df466723] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:31.784 INFO 11 --- [pool-ocp-schedules-9,,1eb9df466723] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:31.931 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131862, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:31.933 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131862, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:02:31.929+08:00, endTime=null} 2023-05-15 14:02:31.957 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131863, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:31.965 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131863, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:02:31.941+08:00, endTime=null} 2023-05-15 14:02:31.980 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131862, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116264, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:31.928+08:00, sub_task_instance_id=131862}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:02:31.990 INFO 11 --- [pool-ocp-schedules-2,,666657706a87] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:02:31.992 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131861, operation:EXECUTE, state: RUNNING 2023-05-15 14:02:31.992 INFO 11 --- [pool-ocp-schedules-2,,666657706a87] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:02:31.998 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,3c4861f7740d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131863, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116265, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:31.941+08:00, sub_task_instance_id=131863}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:02:31.999 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,3c4861f7740d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:02:32.011 INFO 11 --- [pool-ocp-schedules-3,,23bc86e7f46f] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:02:32.014 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131861, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:02:31.986+08:00, endTime=null} 2023-05-15 14:02:32.015 INFO 11 --- [pool-ocp-schedules-3,,23bc86e7f46f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:02:32.015 INFO 11 --- [pool-ocp-schedules-10,,22a6a6b9655f] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:02:32.017 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.017 INFO 11 --- [pool-ocp-schedules-10,,22a6a6b9655f] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:02:32.017 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.017 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.018 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.021 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.021 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.022 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.023 INFO 11 --- [ForkJoinPool-66012-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:02:32.024 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:02:32.024 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:02:32.025 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:02:32.025 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:02:32.033 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:02:32.037 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:02:32.046 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.046 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.047 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.047 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.047 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.047 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.047 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.048 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.048 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.048 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.049 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:02:32.049 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.049 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.049 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.052 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.052 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.053 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:02:32.053 INFO 11 --- [pool-3243800-thread-4,fb27d19334114ed7,8aaf3d5ee66a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:02:32.054 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.054 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.054 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.055 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.060 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.060 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.060 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.060 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.061 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.061 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.061 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.062 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:02:32.062 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.062 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.062 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.062 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.062 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.062 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.063 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:02:32.063 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:02:32.066 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.066 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.068 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.069 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.069 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.069 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.070 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.070 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.070 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.071 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.072 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:02:32.073 INFO 11 --- [pool-3243800-thread-3,fb27d19334114ed7,5b88fcbedc9f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:02:32.073 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:02:32.074 INFO 11 --- [pool-3243800-thread-1,fb27d19334114ed7,298f66aabdf9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:02:32.074 INFO 11 --- [ForkJoinPool-66012-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:02:32.075 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:02:32.075 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.076 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.076 INFO 11 --- [pool-3243800-thread-2,fb27d19334114ed7,a2c4d161f26c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:02:32.076 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:02:32.078 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.078 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.078 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:02:32.078 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:02:32.080 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.081 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.081 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.095 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.096 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.096 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.097 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131861, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116264, task_operation=execute, latest_execution_start_time=2023-05-15T14:02:31.986+08:00, sub_task_instance_id=131861}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:02:32.098 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.103 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.104 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.104 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:02:32.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.110 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.111 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.111 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:02:32.112 INFO 11 --- [pool-3243800-thread-6,fb27d19334114ed7,123fe4d035e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:02:32.115 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.115 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.116 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.120 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.120 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.121 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:02:32.124 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.124 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.124 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.125 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:02:32.126 INFO 11 --- [pool-3243800-thread-5,fb27d19334114ed7,0b058c86720a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:02:32.126 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,3c4861f7740d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:02:32.131 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,3c4861f7740d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131863, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:32.134 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.134 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.135 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.135 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.135 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.135 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.135 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.135 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.135 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.136 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.136 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.136 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:02:32.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.136 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.137 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.137 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.137 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:02:32.137 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.137 WARN 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:02:32.138 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.138 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.138 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:02:32.139 INFO 11 --- [ForkJoinPool-66014-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:02:32.139 INFO 11 --- [ForkJoinPool-66014-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:02:32.139 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:02:32.143 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.143 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.143 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.146 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.146 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:02:32.148 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:02:32.153 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.154 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.155 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.155 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.155 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.157 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.157 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.157 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.157 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.159 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.159 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.160 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:02:32.162 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.162 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.162 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:02:32.165 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:02:32.166 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.166 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.167 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.168 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.169 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.169 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.169 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.169 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.171 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.171 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.171 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.177 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.177 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.177 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.180 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:02:32.180 INFO 11 --- [ForkJoinPool-66014-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:02:32.180 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.182 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.182 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:02:32.186 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.186 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.192 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.192 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.193 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:02:32.197 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.197 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:02:32.202 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.202 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.202 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:02:32.204 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.205 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.205 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.205 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.206 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.210 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.210 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.211 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.212 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.212 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:02:32.212 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.212 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.213 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:02:32.228 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.228 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.229 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:02:32.234 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.234 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:02:32.240 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:02:32.250 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.251 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.252 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.252 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.253 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.253 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:02:32.257 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.257 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.258 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.263 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.264 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.264 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.264 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.264 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.266 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.266 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.266 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.271 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:02:32.279 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.280 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.280 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.281 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.281 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.281 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.281 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.281 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.282 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.284 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.287 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:02:32.288 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.289 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.289 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.289 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.290 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.290 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:02:32.294 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.294 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.295 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.306 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.307 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.307 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.307 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.308 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.309 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.309 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.310 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.313 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:02:32.313 INFO 11 --- [ForkJoinPool-66014-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:02:32.324 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.324 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.325 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.325 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.325 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.325 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.326 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.327 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.327 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.327 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.328 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.333 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.333 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.334 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.349 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.352 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.352 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:02:32.352 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.353 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.354 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.355 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.355 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.366 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.368 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:02:32.368 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:02:32.368 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:02:32.370 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.372 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.373 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,25cae61a4fb5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131862, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:32.373 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.373 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.374 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.394 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.395 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.395 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.395 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.395 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.396 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.396 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.397 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:02:32.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.411 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.412 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.413 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:02:32.414 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:02:32.418 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:02:32.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:02:32.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.445 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.445 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.445 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:02:32.448 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.449 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.449 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:02:32.465 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.468 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:02:32.469 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.469 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.469 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:02:32.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.480 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:02:32.480 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.480 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.481 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:02:32.490 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.492 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:02:32.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:02:32.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:02:32.493 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:02:32.501 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:02:32.503 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:02:32.503 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:02:32.503 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:02:32.508 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3fce0f74f85a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131861, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:02:32.690 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:32.691 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:32.751 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116264, status:SUCCESS 2023-05-15 14:02:32.752 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116264, status:SUCCESS 2023-05-15 14:02:32.765 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116265, status:SUCCESS 2023-05-15 14:02:32.766 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116265, status:SUCCESS 2023-05-15 14:02:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:34.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:34.828588+08:00, lastProcessTime=2023-05-15T14:02:24.827669+08:00 2023-05-15 14:02:34.855 INFO 11 --- [alarm-detect-13,,598e2e878af0] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130534 ~ 1684130544 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130540=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:34.890 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:34.828588+08:00 2023-05-15 14:02:35.691 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:35.692 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:35.931 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:35.938 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:35.943 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:35.950 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:36.917 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:37.338 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c1d2f33098d047dc,9d5913d2b6ad] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=c1d2f33098d047dc, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:02:37.340 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c1d2f33098d047dc,9d5913d2b6ad] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=c1d2f33098d047dc, duration=2 ms] 2023-05-15 14:02:37.753 INFO 11 --- [pool-ocp-schedules-11,,59381a901e71] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:38.260 INFO 11 --- [pool-overview-container-5,,5017862e6ff4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:38.508 INFO 11 --- [pool-scanner--15970743-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:38.692 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:38.694 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:41.694 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:41.695 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:41.996 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,b298c4c51bcf4ee0,39e4656d4733] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=b298c4c51bcf4ee0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:42.001 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,b298c4c51bcf4ee0,39e4656d4733] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=b298c4c51bcf4ee0, duration=6 ms] 2023-05-15 14:02:42.344 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:44.695 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:44.696 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:44.831860+08:00, lastProcessTime=2023-05-15T14:02:34.828588+08:00 2023-05-15 14:02:44.888 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:44.831860+08:00 2023-05-15 14:02:44.894 INFO 11 --- [alarm-detect-8,,4ed6dab361a2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130544 ~ 1684130554 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.768, values={1684130550=245.768}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:45.473 INFO 11 --- [pool-ocp-schedules-13,,b32c339d91a6] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:02:45.936 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:45.943 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:45.946 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:45.952 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:46.804 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802} 2023-05-15 14:02:46.843 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.844 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.845 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.846 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.847 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.848 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.853 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.855 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.856 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.857 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.858 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:46.743, end=2023-05-15T14:00:46.802},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:02:46.858 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:02:46.858 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:02:46.920 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:47.697 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:47.698 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:47.775 INFO 11 --- [pool-ocp-schedules-8,,3bfb15d0d32a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:48.260 INFO 11 --- [pool-overview-container-2,,f1b4f2020dc8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:48.509 INFO 11 --- [pool-scanner--15970843-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:49.531 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,23daad04d9b74ef0,d977c64963e2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=23daad04d9b74ef0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:49.536 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,23daad04d9b74ef0,d977c64963e2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=23daad04d9b74ef0, duration=5 ms] 2023-05-15 14:02:50.698 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:50.699 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:51.789 INFO 11 --- [pool-ocp-schedules-10,,f98e6f0ba7c8] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:51.789 INFO 11 --- [pool-ocp-schedules-10,,f98e6f0ba7c8] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:02:51.800 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:51.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:51.804 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:52.801 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,1d9450fabff84b2b,c47989ce2166] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=1d9450fabff84b2b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:52.807 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,1d9450fabff84b2b,c47989ce2166] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=1d9450fabff84b2b, duration=6 ms] 2023-05-15 14:02:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,c32c91894f0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40165,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":80141} 2023-05-15 14:02:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,c32c91894f0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:02:53.699 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:53.700 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:02:54.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:02:54.831624+08:00, lastProcessTime=2023-05-15T14:02:44.831860+08:00 2023-05-15 14:02:54.865 INFO 11 --- [alarm-detect-13,,ce8dc8b9947f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130554 ~ 1684130564 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.813, values={1684130560=245.813}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:02:54.907 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:02:54.831624+08:00 2023-05-15 14:02:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,62b7133db80c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:02:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,6afca02a935a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:02:55.938 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:55.944 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:55.948 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:55.955 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:56.315 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ed4334078f474d3a,8898540c7574] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=ed4334078f474d3a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:02:56.320 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ed4334078f474d3a,8898540c7574] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=ed4334078f474d3a, duration=5 ms] 2023-05-15 14:02:56.701 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:56.702 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:02:56.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:57.347 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:02:57.798 INFO 11 --- [pool-ocp-schedules-10,,5f0ce7bca0ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:02:58.261 INFO 11 --- [pool-overview-container-4,,52c33749d9a8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:02:58.509 INFO 11 --- [pool-scanner--15970937-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:02:59.702 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:02:59.703 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:03:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130580 2023-05-15 14:03:00.012 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:03:00.012 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:03:00.012 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:03:00.012 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:03:00.018 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:03:00.018 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:03:00.018 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:03:00.018 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:03:00.052 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:03:00.056 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:03:00.065 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:03:00.065 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:03:00.065 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:03:00.065 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:03:00.065 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:03:00.072 WARN 11 --- [pool-ocp-schedules-20,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:03:00.072 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:03:00.574 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:03:00.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:03:00.0, currentDate=Mon May 15 14:03:00 CST 2023 2023-05-15 14:03:00.892 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:03:00.0} 2023-05-15 14:03:00.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:03:00.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:00.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:00.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:03:00.0, currentDate=Mon May 15 14:03:00 CST 2023 2023-05-15 14:03:00.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:03:00.0} 2023-05-15 14:03:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:03:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:00.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:00.919 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:03:00.0, currentDate=Mon May 15 14:03:00 CST 2023 2023-05-15 14:03:00.919 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:03:00.0} 2023-05-15 14:03:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:03:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:00.933 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:00.938 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:31.0, nextRun=2023-05-15 14:03:00.0, currentDate=Mon May 15 14:03:00 CST 2023 2023-05-15 14:03:00.938 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:02:31.0, nextRunTime=2023-05-15 14:03:00.0} 2023-05-15 14:03:00.942 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:03:00.942 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:00.942 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:00.952 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:03:00.0, currentDate=Mon May 15 14:03:00 CST 2023 2023-05-15 14:03:00.952 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:03:00.0} 2023-05-15 14:03:00.956 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:03:00.956 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:00.956 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:01.041 WARN 11 --- [pool-ocp-schedules-20,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:03:01.042 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.024094472S 2023-05-15 14:03:01.042 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:03:01.160 INFO 11 --- [pool-ocp-schedules-20,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:03:01.231 INFO 11 --- [pool-ocp-schedules-20,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:03:01.268 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:03:01.269 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:03:01.269 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:03:01.269 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:03:01.269 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:03:01.269 INFO 11 --- [pool-ocp-schedules-20,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:03:01.279 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:03:01.449 INFO 11 --- [pool-ocp-schedules-20,,] o.s.boot.SpringApplication : Started application in 1.437 seconds (JVM running for 600237.423) 2023-05-15 14:03:01.449 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.4373046S 2023-05-15 14:03:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,f3fa48d26a80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130640, interval=60 2023-05-15 14:03:01.515 INFO 11 --- [pool-prometheus-minute-schedule-2,,1ffd2997c43a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130640, interval=60 2023-05-15 14:03:01.533 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131866, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:01.535 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131866, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:03:01.510+08:00, endTime=null} 2023-05-15 14:03:01.542 INFO 11 --- [pool-prometheus-minute-schedule-2,,1ffd2997c43a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:03:01.542 INFO 11 --- [pool-prometheus-minute-schedule-2,,1ffd2997c43a] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130581, count=6, elapsed=1682446450961 2023-05-15 14:03:01.542 INFO 11 --- [pool-prometheus-minute-schedule-1,,f3fa48d26a80] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:03:01.543 INFO 11 --- [pool-prometheus-minute-schedule-1,,f3fa48d26a80] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130581, count=8, elapsed=1682446450962 2023-05-15 14:03:01.544 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131867, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:01.545 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131867, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:03:01.542+08:00, endTime=null} 2023-05-15 14:03:01.550 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ee959f9cd843] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131866, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116268, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:01.510+08:00, sub_task_instance_id=131866}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:01.554 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131864, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:01.555 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131864, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:03:01.551+08:00, endTime=null} 2023-05-15 14:03:01.561 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6b9219e4ba1d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131867, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116269, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:01.542+08:00, sub_task_instance_id=131867}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:01.561 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6b9219e4ba1d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:03:01.564 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131865, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:01.565 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131865, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:03:01.562+08:00, endTime=null} 2023-05-15 14:03:01.568 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ee959f9cd843] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:03:01.568 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ee959f9cd843] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:03:01.570 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,5fc2a076ea19] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131864, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116266, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:01.551+08:00, sub_task_instance_id=131864}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:01.571 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ee959f9cd843] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:03:01.574 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131868, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:01.575 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,ee959f9cd843] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131866, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:01.575 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:03:01.575 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:03:01.575 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:03:01.575 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:03:01.575 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:03:01.575 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131868, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:03:01.572+08:00, endTime=null} 2023-05-15 14:03:01.578 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:03:01.578 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:03:01.578 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:03:01.579 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:03:01.579 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:03:01.579 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:03:01.579 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,5fc2a076ea19] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131864, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:01.579 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:03:01.579 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:03:01.579 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:03:01.580 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67a9adc80087] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131865, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116267, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:01.562+08:00, sub_task_instance_id=131865}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:01.581 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:03:01.581 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.590 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.591 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.592 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.592 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.592 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.592 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.594 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,88ff5e80fc7f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131868, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116270, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:01.572+08:00, sub_task_instance_id=131868}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:01.596 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.597 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.597 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.597 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.598 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.598 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.598 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67a9adc80087] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:03:01.598 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.598 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:01.599 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67a9adc80087] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:03:01.603 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,67a9adc80087] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131865, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:01.603 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:03:01.603 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.603 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.603 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:01.604 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.604 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.605 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:03:01.605 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:03:01.605 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:03:01.605 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:03:01.606 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:03:01.606 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:03:01.607 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,88ff5e80fc7f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131868, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:01.607 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:03:01.607 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:03:01.608 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:03:01.608 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.608 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.609 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:03:01.610 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:01.610 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:01.611 INFO 11 --- [pool-3243967-thread-1,fb27d19334114ed7,ce481e450a5a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:03:01.611 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:03:01.611 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:03:01.611 INFO 11 --- [pool-3243967-thread-3,fb27d19334114ed7,ecd4a74456e4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:03:01.613 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:03:01.613 INFO 11 --- [pool-3243967-thread-2,fb27d19334114ed7,89b0559c9203] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:03:01.615 INFO 11 --- [pool-3243967-thread-4,fb27d19334114ed7,67ce5258a4e7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:03:01.618 INFO 11 --- [pool-3243967-thread-5,fb27d19334114ed7,27058e92ee62] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:03:01.619 INFO 11 --- [pool-3243967-thread-6,fb27d19334114ed7,052e68b79c18] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:03:01.620 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6b9219e4ba1d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:03:01.621 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:03:01.624 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:03:01.625 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6b9219e4ba1d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131867, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:01.630 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:03:01.632 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:03:01.640 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:03:01.682 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:03:01.691 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:03:01.691 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:03:01.691 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:03:01.691 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:03:01.694 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:03:01.705 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:03:02.055 INFO 11 --- [pool-ocp-schedules-20,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:03:02.393 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116266, status:SUCCESS 2023-05-15 14:03:02.396 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116266, status:SUCCESS 2023-05-15 14:03:02.405 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116267, status:SUCCESS 2023-05-15 14:03:02.405 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116267, status:SUCCESS 2023-05-15 14:03:02.414 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116268, status:SUCCESS 2023-05-15 14:03:02.415 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116268, status:SUCCESS 2023-05-15 14:03:02.425 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116269, status:SUCCESS 2023-05-15 14:03:02.426 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116269, status:SUCCESS 2023-05-15 14:03:02.437 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116270, status:SUCCESS 2023-05-15 14:03:02.438 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116270, status:SUCCESS 2023-05-15 14:03:02.512 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=971 2023-05-15 14:03:02.512 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=971 2023-05-15 14:03:02.514 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.514 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=976 2023-05-15 14:03:02.514 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.515 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=976 2023-05-15 14:03:02.516 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=970 2023-05-15 14:03:02.516 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=970 2023-05-15 14:03:02.523 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=969 2023-05-15 14:03:02.523 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=969 2023-05-15 14:03:02.523 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=969 2023-05-15 14:03:02.523 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=969 2023-05-15 14:03:02.533 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=962 2023-05-15 14:03:02.533 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=962 2023-05-15 14:03:02.553 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=967 2023-05-15 14:03:02.553 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=967 2023-05-15 14:03:02.621 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.621 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.622 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.622 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.627 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.627 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.629 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.629 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.634 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.634 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.652 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.652 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.667 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.667 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.668 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.668 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.703 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:02.704 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:02.715 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.715 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.718 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.718 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.722 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.722 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.731 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.731 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.732 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.732 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.748 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.748 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.766 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.766 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.771 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.772 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.809 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.810 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.810 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.810 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.821 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.821 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.821 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.821 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.837 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.837 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.841 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.842 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.864 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.865 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.875 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.875 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.909 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.909 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.909 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.909 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.915 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.915 INFO 11 --- [pool-metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.916 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=19, affectRows=19 2023-05-15 14:03:02.917 INFO 11 --- [pool-metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=19, validCount=19 2023-05-15 14:03:02.919 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.919 INFO 11 --- [pool-metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.927 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=93, affectRows=93 2023-05-15 14:03:02.927 INFO 11 --- [pool-metric-archive-minute-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=93, validCount=93 2023-05-15 14:03:02.934 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.934 INFO 11 --- [pool-metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.936 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.936 INFO 11 --- [pool-metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.944 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.944 INFO 11 --- [pool-metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:02.964 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2023-05-15 14:03:02.964 INFO 11 --- [pool-metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1684126800, nodeCount=1000, validCount=1000 2023-05-15 14:03:04.714 INFO 11 --- [pool-ocp-schedules-7,,fb10176350f7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:03:04.721 INFO 11 --- [pool-ocp-schedules-7,,fb10176350f7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:03:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:04.836 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:04.835676+08:00, lastProcessTime=2023-05-15T14:02:54.831624+08:00 2023-05-15 14:03:04.864 INFO 11 --- [alarm-detect-15,,815dcce7320e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130564 ~ 1684130574 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.858, values={1684130570=245.858}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:04.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:04.835676+08:00 2023-05-15 14:03:05.113 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:05.0, nextRun=2023-05-15 14:03:05.0, currentDate=Mon May 15 14:03:05 CST 2023 2023-05-15 14:03:05.113 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:02:05.0, nextRunTime=2023-05-15 14:03:05.0} 2023-05-15 14:03:05.117 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:03:05.117 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:05.117 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:05.642 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131869, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:05.644 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131869, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:03:05.639+08:00, endTime=null} 2023-05-15 14:03:05.666 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,986e74dcfd77] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131869, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116271, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:05.638+08:00, sub_task_instance_id=131869}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:05.692 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,0c45762d0ee7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:03:05.693 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,556dd9516c29] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:03:05.693 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,011fc8eca300] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:03:05.705 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:05.706 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:05.742 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,0c45762d0ee7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:03:05.748 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,011fc8eca300] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:03:05.756 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,556dd9516c29] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:03:05.769 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,986e74dcfd77] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131869, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:05.942 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:05.947 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:05.950 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:05.957 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:06.531 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116271, status:SUCCESS 2023-05-15 14:03:06.533 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116271, status:SUCCESS 2023-05-15 14:03:06.927 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:07.594 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,0ab2ffbf26f04335,cb4acb94f4e2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=0ab2ffbf26f04335, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:07.600 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,0ab2ffbf26f04335,cb4acb94f4e2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=0ab2ffbf26f04335, duration=7 ms] 2023-05-15 14:03:07.818 INFO 11 --- [pool-ocp-schedules-19,,43b81d1666e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:08.264 INFO 11 --- [pool-overview-container-1,,814cce96d65c] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:03:08.264 INFO 11 --- [pool-overview-container-1,,814cce96d65c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:08.426 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,09e76224aae54a4f,3d5854501455] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=09e76224aae54a4f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:08.432 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,09e76224aae54a4f,3d5854501455] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=09e76224aae54a4f, duration=7 ms] 2023-05-15 14:03:08.510 INFO 11 --- [pool-scanner--15971030-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:03:08.706 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:08.707 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:09.945 INFO 11 --- [pool-ocp-schedules-14,,3b8077801af6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40033} 2023-05-15 14:03:09.945 INFO 11 --- [pool-ocp-schedules-14,,3b8077801af6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:03:10.203 INFO 11 --- [pool-ocp-schedules-20,,c8e4f400101c] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:03:11.369 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,e557a1c1dd534c71,ccc593bf2715] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=e557a1c1dd534c71, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:03:11.370 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,e557a1c1dd534c71,ccc593bf2715] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e557a1c1dd534c71, duration=2 ms] 2023-05-15 14:03:11.707 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:11.708 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:11.795 INFO 11 --- [pool-ocp-schedules-2,,a255ab8d3300] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:11.795 INFO 11 --- [pool-ocp-schedules-2,,a255ab8d3300] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:12.350 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:12.848 INFO 11 --- [pool-ocp-schedules-2,,fd756896003a] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:03:12.848 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:03:12.848 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:03:12.848 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:03:12.848 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:03:12.848 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:03:12.855 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:03:12.855 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:03:12.855 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:03:12.856 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:03:12.856 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:03:12.881 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:03:12.883 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:03:12.883 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:03:12.884 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:03:12.884116703+08:00 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:03:12.884+08:00 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:03:12.884840910+08:00 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:03:12.885+08:00 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:03:12.885342245+08:00 2023-05-15 14:03:12.885 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:03:12.885+08:00 2023-05-15 14:03:12.887 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:03:12.886904953+08:00 2023-05-15 14:03:12.887 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:03:12.887+08:00 2023-05-15 14:03:12.888 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:03:12.891 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:03:12.890633252+08:00 2023-05-15 14:03:12.891 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:03:12.891+08:00 2023-05-15 14:03:12.895 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:03:12.885+08:00 and status AVAILABLE 2023-05-15 14:03:12.899 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:03:12.887+08:00 and status AVAILABLE 2023-05-15 14:03:12.900 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:03:12.900 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:03:12.885+08:00 and status AVAILABLE 2023-05-15 14:03:12.922 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130592 0 0 , error= 2023-05-15 14:03:12.923 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:03:12.923+08:00 2023-05-15 14:03:12.924 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:03:12.924 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:03:12.884+08:00 and status AVAILABLE 2023-05-15 14:03:12.930 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:03:12.930 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:03:12.923+08:00 and status AVAILABLE 2023-05-15 14:03:12.930 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:03:12.891+08:00 and status AVAILABLE 2023-05-15 14:03:12.933 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:03:12.951 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130592 0 0 , error= 2023-05-15 14:03:12.952 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:03:12.952+08:00 2023-05-15 14:03:12.953 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:03:12.972 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130592 0 0 , error= 2023-05-15 14:03:12.973 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:03:12.973+08:00 2023-05-15 14:03:12.973 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:03:12.952+08:00 and status AVAILABLE 2023-05-15 14:03:12.973 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:03:12.993 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130592 0 0 , error= 2023-05-15 14:03:12.993 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:03:12.993+08:00 2023-05-15 14:03:12.994 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:03:12.994 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:03:12.973+08:00 and status AVAILABLE 2023-05-15 14:03:12.995 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:03:12.996 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:03:12.998 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:03:12.993+08:00 and status AVAILABLE 2023-05-15 14:03:13.001 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:03:13.006 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:03:13.007 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:03:13.012 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:03:13.027 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:03:13.034 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:03:13.036 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:03:13.039 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:03:13.079 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:03:13.086 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:03:13.086 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:03:13.093 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:03:13.093 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:03:13.100 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:03:13.101 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:03:13.106 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:03:13.121 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:03:13.126 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:03:13.127 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:03:13.129 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:03:13.177 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:03:13.178 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:03:13.180 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:03:13.188 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:03:13.192 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:03:13.193 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:03:13.195 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:03:13.201 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:03:13.210 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:03:13.211 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:03:13.212 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:03:13.219 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:03:13.257 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:03:13.262 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:03:13.268 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:03:13.274 INFO 11 --- [pool-Check host status7,,a5f396b63341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:03:13.278 INFO 11 --- [pool-Check host status6,,2ad25518c619] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:03:13.283 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:03:13.284 INFO 11 --- [pool-Check host status4,,1bdce8bd38df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:03:13.297 INFO 11 --- [pool-Check host status29,,df351d545b47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:03:13.880 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130593 0 0 , error= 2023-05-15 14:03:13.883 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:03:13.882+08:00 2023-05-15 14:03:13.888 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:03:13.882+08:00 and status AVAILABLE 2023-05-15 14:03:13.891 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:03:13.902 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:03:13.939 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:03:14.022 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:03:14.038 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:03:14.060 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:03:14.148 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:03:14.162 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:03:14.179 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:03:14.255 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:03:14.269 INFO 11 --- [pool-Check host status21,,0798fd8bd0e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:03:14.292 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:03:14.292 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:03:14.292 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:03:14.292 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:03:14.292 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:03:14.299 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:03:14.299 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:03:14.299 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:03:14.299 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:03:14.299 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:03:14.324 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:03:14.324 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:03:14.324 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:03:14.325 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:03:14.328 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:03:14.333 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130520, nodeCount=690, affectRows=690 2023-05-15 14:03:14.335 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130520, nodeCount=1000, affectRows=1000 2023-05-15 14:03:14.336 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130520, nodeCount=1000, affectRows=1000 2023-05-15 14:03:14.338 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130520, nodeCount=690, validCount=690 2023-05-15 14:03:14.341 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130520, nodeCount=1000, validCount=1000 2023-05-15 14:03:14.342 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:03:14.342 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:03:14.343 INFO 11 --- [pool-Check host status23,,59327f84593b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:03:14.343 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130520, nodeCount=1000, validCount=1000 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status5,,0f42ea10a8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:03:14.344 INFO 11 --- [pool-Check host status27,,adc812238ea6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:03:14.346 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:03:14.346 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:03:14.346 INFO 11 --- [pool-Check host status20,,16e196b08f63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:03:14.348 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:03:14.348 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:03:14.348 INFO 11 --- [pool-Check host status11,,9df1c76db46b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:03:14.708 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:14.709 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:14.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:14.830580+08:00, lastProcessTime=2023-05-15T14:03:04.835676+08:00 2023-05-15 14:03:14.856 INFO 11 --- [alarm-detect-15,,ade65ff0605a] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130574 ~ 1684130584 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130580=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:14.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:14.830580+08:00 2023-05-15 14:03:15.894 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,d39a1ad8fa204cfb,34325420260b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=d39a1ad8fa204cfb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:15.899 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,d39a1ad8fa204cfb,34325420260b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=d39a1ad8fa204cfb, duration=6 ms] 2023-05-15 14:03:15.946 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:15.949 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:15.952 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:15.959 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:16.930 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:17.541 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,ad45a8f75c314721,7f3e06a878cc] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&clusterId=2, client=192.168.80.229, traceId=ad45a8f75c314721, method=PaginatedResponse com.oceanbase.ocp.server.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 2000, sort: id: ASC],NORMAL,TENANT,2,] 2023-05-15 14:03:17.618 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:03:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.655 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:17.662 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,ad45a8f75c314721,7f3e06a878cc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=192.168.80.229, traceId=ad45a8f75c314721, duration=124 ms] 2023-05-15 14:03:17.684 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:17.686 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:17.687 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:17.687 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.688 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:17.693 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:17.693 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.694 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:17.697 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:03:17.697 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:03:17.697 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:03:17.697 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:03:17.703 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:03:17.703 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.703 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:03:17.703 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.704 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:03:17.704 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:17.704 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:03:17.705 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:03:17.705 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:03:17.709 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:17.710 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:17.729 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:00:48, start=2023-05-15T13:58:17, end=2023-05-15T14:00:47, updates=100, granule=120000ms, delay=151000ms 2023-05-15 14:03:17.731 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:00:48, target=2023-05-15T14:00:47, granule=120000ms, delay=1000ms) 2023-05-15 14:03:17.731 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:51:17, granule=600000ms, delay=570000ms) 2023-05-15 14:03:17.732 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:01:02, start=2023-05-15T13:58:17, end=2023-05-15T14:00:47, updates=104, granule=120000ms, delay=165000ms 2023-05-15 14:03:17.732 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:00:48, start=2023-05-15T13:58:17, end=2023-05-15T14:00:47, updates=154, granule=120000ms, delay=151000ms 2023-05-15 14:03:17.734 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:01:02, target=2023-05-15T14:00:47, granule=120000ms, delay=15000ms) 2023-05-15 14:03:17.734 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:53:17, granule=600000ms, delay=450000ms) 2023-05-15 14:03:17.734 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:00:48, target=2023-05-15T14:00:47, granule=120000ms, delay=1000ms) 2023-05-15 14:03:17.734 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:51:17, granule=600000ms, delay=570000ms) 2023-05-15 14:03:17.734 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=116ms, period=120000ms 2023-05-15 14:03:17.829 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,86f06d4af470457b,232b52997c7b] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=sql_all_rt%2Csql_select_rt%2Csql_insert_rt%2Csql_update_rt%2Csql_replace_rt%2Csql_delete_rt%2Csql_other_rt&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=86f06d4af470457b, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:17.841 INFO 11 --- [pool-ocp-schedules-13,,c6b2ec2bea55] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:17.844 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a25515b0bff44101,e230e2257995] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=sql_local_time%2Csql_remote_time%2Csql_distributed_time&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=a25515b0bff44101, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,sql_local_time,sql_remote_time,sql_distributed_time,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:17.863 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,2e80807771664e27,e42a66fc64ff] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=sql_local_count%2Csql_remote_count%2Csql_distributed_count&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=2e80807771664e27, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,sql_local_count,sql_remote_count,sql_distributed_count,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:17.873 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,4f45e7a116b34cf0,e100a722f9f7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=sql_all_count%2Csql_select_count%2Csql_insert_count%2Csql_update_count%2Csql_replace_count%2Csql_delete_count%2Csql_other_count&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=4f45e7a116b34cf0, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:17.885 INFO 11 --- [pool-scanner--15971146-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.886 INFO 11 --- [pool-scanner--15971133-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.893 INFO 11 --- [pool-scanner--15971132-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.893 INFO 11 --- [pool-scanner--15971142-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.893 INFO 11 --- [pool-scanner--15971137-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.894 INFO 11 --- [pool-scanner--15971140-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.895 INFO 11 --- [pool-scanner--15971138-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.895 INFO 11 --- [pool-scanner--15971131-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.895 INFO 11 --- [pool-scanner--15971135-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.895 INFO 11 --- [pool-scanner--15971148-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.896 INFO 11 --- [pool-scanner--15971136-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.896 INFO 11 --- [pool-scanner--15971134-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.896 INFO 11 --- [pool-scanner--15971149-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.897 INFO 11 --- [pool-scanner--15971130-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.898 INFO 11 --- [pool-scanner--15971144-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.904 INFO 11 --- [pool-scanner--15971139-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.905 INFO 11 --- [pool-scanner--15971145-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.906 INFO 11 --- [pool-scanner--15971143-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.906 INFO 11 --- [pool-scanner--15971147-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.909 INFO 11 --- [pool-scanner--15971169-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.911 INFO 11 --- [pool-scanner--15971171-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.912 INFO 11 --- [pool-scanner--15971156-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.912 INFO 11 --- [pool-scanner--15971153-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.913 INFO 11 --- [pool-scanner--15971141-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.915 INFO 11 --- [pool-scanner--15971159-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.915 INFO 11 --- [pool-scanner--15971157-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.916 INFO 11 --- [pool-scanner--15971160-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.916 INFO 11 --- [pool-scanner--15971164-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.916 INFO 11 --- [pool-scanner--15971152-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.916 INFO 11 --- [pool-scanner--15971158-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.917 INFO 11 --- [pool-scanner--15971165-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.917 INFO 11 --- [pool-scanner--15971161-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.917 INFO 11 --- [pool-scanner--15971155-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.919 INFO 11 --- [pool-scanner--15971154-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.920 INFO 11 --- [pool-scanner--15971163-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.921 INFO 11 --- [pool-scanner--15971168-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.921 INFO 11 --- [pool-scanner--15971166-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.921 INFO 11 --- [pool-scanner--15971167-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.922 INFO 11 --- [pool-scanner--15971162-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.923 INFO 11 --- [pool-scanner--15971151-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.925 INFO 11 --- [pool-scanner--15971170-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.925 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,2e80807771664e27,e42a66fc64ff] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=2e80807771664e27, duration=63 ms] 2023-05-15 14:03:17.927 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a25515b0bff44101,e230e2257995] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=a25515b0bff44101, duration=86 ms] 2023-05-15 14:03:17.925 INFO 11 --- [pool-scanner--15971150-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:17.946 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,4f45e7a116b34cf0,e100a722f9f7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=4f45e7a116b34cf0, duration=75 ms] 2023-05-15 14:03:17.961 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,86f06d4af470457b,232b52997c7b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=86f06d4af470457b, duration=134 ms] 2023-05-15 14:03:18.261 INFO 11 --- [pool-overview-container-6,,532333918992] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:18.341 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,518c8674f8094157,fa8a78e90ae7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=wait_event_count%2Csystem_event_internal_total_waits%2Csystem_event_sync_rpc_total_waits%2Csystem_event_row_lock_wait_total_waits%2Csystem_event_io_total_waits%2Csystem_event_latch_total_waits%2Csystem_event_other_total_waits&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=518c8674f8094157, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,wait_event_count,system_event_internal_total_waits,system_event_sync_rpc_total_waits,system_event_row_lock_wait_total_waits,system_event_io_total_waits,system_event_latch_total_waits,system_event_other_total_waits,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:18.358 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,ed8ae6188bba4f62,7e5c5043182d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A03%3A17%2B08%3A00&endTime=2023-05-15T14%3A03%3A17%2B08%3A00&metrics=active_session%2Call_session&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=ed8ae6188bba4f62, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:03:17+08:00,2023-05-15T14:03:17+08:00,active_session,all_session,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:18.364 INFO 11 --- [pool-scanner--15971173-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.364 INFO 11 --- [pool-scanner--15971174-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.364 INFO 11 --- [pool-scanner--15971172-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.366 INFO 11 --- [pool-scanner--15971175-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.371 INFO 11 --- [pool-scanner--15971176-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.371 INFO 11 --- [pool-scanner--15971178-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.372 INFO 11 --- [pool-scanner--15971177-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.377 INFO 11 --- [pool-scanner--15971180-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.378 INFO 11 --- [pool-scanner--15971179-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=171, nodesFilledCount=57 2023-05-15 14:03:18.379 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,518c8674f8094157,fa8a78e90ae7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=518c8674f8094157, duration=40 ms] 2023-05-15 14:03:18.383 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,ed8ae6188bba4f62,7e5c5043182d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=ed8ae6188bba4f62, duration=28 ms] 2023-05-15 14:03:18.514 INFO 11 --- [pool-scanner--15971181-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:20.005 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,6485bda371e44341,85808d12fdd3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames?groupBy=tablegroup_name, client=192.168.80.229, traceId=6485bda371e44341, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrendsSubLevelNames(Long,Long,String,String), args=2,6,tablegroup_name,] 2023-05-15 14:03:20.005 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,d1c759919b8642ae,b80a04b288e8] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames?groupBy=database_name, client=192.168.80.229, traceId=d1c759919b8642ae, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrendsSubLevelNames(Long,Long,String,String), args=2,6,database_name,] 2023-05-15 14:03:20.005 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,6485bda371e44341,85808d12fdd3] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[], labels={}, interval=null, minStep=null, groupBy=[tablegroup_name], limit=null, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:03:20.005 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,d1c759919b8642ae,b80a04b288e8] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=null, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:03:20.007 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,7ec5e00daa5b4769,0d448f30bdf9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=7ec5e00daa5b4769, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:03:20.007 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,7ec5e00daa5b4769,0d448f30bdf9] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:03:20.009 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,372ebc1b1dfc4ba8,3b0b83cecfe7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A03%3A19%2B08%3A00&endTime=2023-05-15T14%3A03%3A19%2B08%3A00&metrics=ob_tenant_data_size%2Cob_tenant_disk_used%2Cob_tenant_disk_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=372ebc1b1dfc4ba8, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:03:19+08:00,2023-05-15T14:03:19+08:00,ob_tenant_data_size,ob_tenant_disk_used,ob_tenant_disk_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:20.009 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,372ebc1b1dfc4ba8,3b0b83cecfe7] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:03:19+08:00, endTime=2023-05-15T14:03:19+08:00, metrics=[ob_tenant_data_size, ob_tenant_disk_used, ob_tenant_disk_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:03:20.012 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0deb8676ec1c48ed,4bb2a737fbea] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A03%3A19%2B08%3A00&endTime=2023-05-15T14%3A03%3A19%2B08%3A00&metrics=ob_tenant_min_cpu%2Cob_tenant_max_cpu_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=0deb8676ec1c48ed, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:03:19+08:00,2023-05-15T14:03:19+08:00,ob_tenant_min_cpu,ob_tenant_max_cpu_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:20.013 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0deb8676ec1c48ed,4bb2a737fbea] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:03:19+08:00, endTime=2023-05-15T14:03:19+08:00, metrics=[ob_tenant_min_cpu, ob_tenant_max_cpu_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:03:20.013 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,edbe1693cb154dca,2d3ed31858a8] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A03%3A19%2B08%3A00&endTime=2023-05-15T14%3A03%3A19%2B08%3A00&metrics=ob_tenant_min_memory%2Cob_tenant_max_memory_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=edbe1693cb154dca, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:03:19+08:00,2023-05-15T14:03:19+08:00,ob_tenant_min_memory,ob_tenant_max_memory_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:03:20.013 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,edbe1693cb154dca,2d3ed31858a8] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:03:19+08:00, endTime=2023-05-15T14:03:19+08:00, metrics=[ob_tenant_min_memory, ob_tenant_max_memory_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:03:20.027 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,372ebc1b1dfc4ba8,3b0b83cecfe7] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:03:20.030 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,edbe1693cb154dca,2d3ed31858a8] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:03:20.036 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0deb8676ec1c48ed,4bb2a737fbea] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:03:20.043 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,d1c759919b8642ae,b80a04b288e8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames client=192.168.80.229, traceId=d1c759919b8642ae, duration=40 ms] 2023-05-15 14:03:20.044 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,7ec5e00daa5b4769,0d448f30bdf9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=7ec5e00daa5b4769, duration=41 ms] 2023-05-15 14:03:20.044 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,6485bda371e44341,85808d12fdd3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames client=192.168.80.229, traceId=6485bda371e44341, duration=41 ms] 2023-05-15 14:03:20.063 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,504a6c6a262b4b9c,460dbfe1a639] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=504a6c6a262b4b9c, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:03:20.063 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,504a6c6a262b4b9c,460dbfe1a639] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:03:20.083 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,504a6c6a262b4b9c,460dbfe1a639] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=504a6c6a262b4b9c, duration=21 ms] 2023-05-15 14:03:20.102 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,2fb77c6c85e94fed,4b15f26c5bc8] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=2fb77c6c85e94fed, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:03:20.102 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,2fb77c6c85e94fed,4b15f26c5bc8] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:03:20.116 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,2fb77c6c85e94fed,4b15f26c5bc8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=2fb77c6c85e94fed, duration=16 ms] 2023-05-15 14:03:20.127 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,372ebc1b1dfc4ba8,3b0b83cecfe7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=372ebc1b1dfc4ba8, duration=120 ms] 2023-05-15 14:03:20.166 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,edbe1693cb154dca,2d3ed31858a8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=edbe1693cb154dca, duration=155 ms] 2023-05-15 14:03:20.168 INFO 11 --- [http-nio-0.0.0.0-8080-exec-58,0deb8676ec1c48ed,4bb2a737fbea] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=0deb8676ec1c48ed, duration=158 ms] 2023-05-15 14:03:20.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:21.0, nextRun=2023-05-15 14:03:20.0, currentDate=Mon May 15 14:03:20 CST 2023 2023-05-15 14:03:20.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:02:21.0, nextRunTime=2023-05-15 14:03:20.0} 2023-05-15 14:03:20.560 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:03:20.560 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:20.560 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:20.710 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:20.711 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:20.866 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131870, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:20.868 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131870, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:03:20.862+08:00, endTime=null} 2023-05-15 14:03:20.890 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ca4250ee02ee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131870, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116272, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:20.862+08:00, sub_task_instance_id=131870}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:20.892 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ca4250ee02ee] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:03:20.893 INFO 11 --- [ForkJoinPool-66017-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:03:20.893 INFO 11 --- [ForkJoinPool-66017-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:03:20.898 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ca4250ee02ee] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131870, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:21.094 INFO 11 --- [pool-scanner--15971193-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:21.114 INFO 11 --- [pool-scanner--15971194-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:21.134 INFO 11 --- [pool-scanner--15971195-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:21.157 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.163 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.202 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.206 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.223 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.228 INFO 11 --- [pool-ocp-schedules-7,,5357b71da789] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:21.412 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/performance/config?id=2&tenantId=6, client=192.168.80.229, traceId=92806f3000e147ce, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getClusterPerfConfig(Long,Long), args=2,6,] 2023-05-15 14:03:21.412 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,c6dbc4f8df754b65,6ba60324ff10] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch?id=2&tenantId=6, client=192.168.80.229, traceId=c6dbc4f8df754b65, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitch(Long,Long), args=2,6,] 2023-05-15 14:03:21.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,412a839d23eb481d,c32be361f5b9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events?null, client=192.168.80.229, traceId=412a839d23eb481d, method=PaginatedResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchEvents(Long,Long,OffsetDateTime,OffsetDateTime,Pageable), args=2,6,Page request [number: 0, size 2000, sort: timestamp: ASC],] 2023-05-15 14:03:21.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,412a839d23eb481d,c32be361f5b9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events client=192.168.80.229, traceId=412a839d23eb481d, duration=20 ms] 2023-05-15 14:03:21.442 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.442 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.442 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.443 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:21.450 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,c6dbc4f8df754b65,6ba60324ff10] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch client=192.168.80.229, traceId=c6dbc4f8df754b65, duration=40 ms] 2023-05-15 14:03:21.454 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/outlines?null, client=192.168.80.229, traceId=9888e379c75546d3, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObOutlineController.getSqlOutline(Long,Long,String,String,OffsetDateTime,OffsetDateTime,Boolean), args=2,6,] 2023-05-15 14:03:21.472 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.472 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.474 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() REGEXP 'OceanBase.*?CE' 2023-05-15 14:03:21.478 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:03:21.478 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:03:21.492 ERROR 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526904) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=526904) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:03:21.493 ERROR 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] com.alibaba.druid.pool.DruidDataSource : {dataSource-93} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526904) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=526904) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:03:21.494 INFO 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] com.alibaba.druid.pool.DruidDataSource : {dataSource-93,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:03:21.494 INFO 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] com.alibaba.druid.pool.DruidDataSource : {dataSource-93} closing ... 2023-05-15 14:03:21.494 INFO 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] com.alibaba.druid.pool.DruidDataSource : {dataSource-93} closed 2023-05-15 14:03:21.494 ERROR 11 --- [obsdk-init-thread-10,9888e379c75546d3,d88ecd560c83] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=526904) Get Location Cache Fail 2023-05-15 14:03:21.497 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526904) Get Location Cache Fail 2023-05-15 14:03:21.497 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526904) Get Location Cache Fail 2023-05-15 14:03:21.498 WARN 11 --- [http-nio-0.0.0.0-8080-exec-37,9888e379c75546d3,9a77a9202c50] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/outlines client=192.168.80.229, traceId=9888e379c75546d3, duration=45 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:03:21.501 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.501 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.502 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2023-05-15 14:03:21.507 ERROR 11 --- [Druid-ConnectionPool-Create-1381789418,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=526907) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=526907) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:03:21.552 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,c0bc6c874f824d20,60955af6b942] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents?startTime=2023-05-15T13%3A33%3A21%2B08%3A00&endTime=2023-05-15T14%3A03%3A21%2B08%3A00, client=192.168.80.229, traceId=c0bc6c874f824d20, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchOffIntervalLimit3(Long,Long,OffsetDateTime,OffsetDateTime), args=2,6,2023-05-15T13:33:21+08:00,2023-05-15T14:03:21+08:00,] 2023-05-15 14:03:21.553 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A33%3A21%2B08%3A00&endTime=2023-05-15T14%3A03%3A21%2B08%3A00&inner=false, client=192.168.80.229, traceId=853dd2a3ac4f4101, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:33:21+08:00,2023-05-15T14:03:21+08:00,false,] 2023-05-15 14:03:21.553 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:03:21.554 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:03:21.571 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,c0bc6c874f824d20,60955af6b942] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents client=192.168.80.229, traceId=c0bc6c874f824d20, duration=20 ms] 2023-05-15 14:03:21.582 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:21.627 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.627 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.628 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:21.635 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:33:21, end=2023-05-15T14:03:21}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:03:21.652 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:33:21, end=2023-05-15T14:03:21}} 2023-05-15 14:03:21.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.703 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.703 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2023-05-15 14:03:21.733 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:03:21.744 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:21.745 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.746 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.746 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.746 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:21.773 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.773 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.774 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:21.793 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116272, status:SUCCESS 2023-05-15 14:03:21.794 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116272, status:SUCCESS 2023-05-15 14:03:21.802 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:21.805 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:21.807 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:21.834 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:03:21.834 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:03:21.841 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,853dd2a3ac4f4101,ba933c194f52] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=853dd2a3ac4f4101, duration=290 ms] 2023-05-15 14:03:21.931 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:21.931 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:21.932 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:03:22.181 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,92806f3000e147ce,0734322f5672] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/performance/config client=192.168.80.229, traceId=92806f3000e147ce, duration=771 ms] 2023-05-15 14:03:23.712 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:23.713 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:24.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:24.827053+08:00, lastProcessTime=2023-05-15T14:03:14.830580+08:00 2023-05-15 14:03:24.958 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:24.827053+08:00 2023-05-15 14:03:24.961 INFO 11 --- [alarm-detect-9,,8d3995d5e62d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130584 ~ 1684130594 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130590=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:24.962 INFO 11 --- [alarm-detect-9,,8d3995d5e62d] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:03:25.014 INFO 11 --- [alarm-detect-4,,295567bb7ff2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130534 ~ 1684130594 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=388977.988, values={1684130580=388977.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:03:25.015 INFO 11 --- [alarm-detect-4,,295567bb7ff2] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130534 ~ 1684130594 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993775.358, values={1684130580=993775.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:25.015 INFO 11 --- [alarm-detect-4,,295567bb7ff2] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:03:25.019 INFO 11 --- [alarm-detect-2,,62fa32874c96] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130535 ~ 1684130595 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=388977.938, values={1684130580=388977.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:03:25.019 INFO 11 --- [alarm-detect-2,,62fa32874c96] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130535 ~ 1684130595 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993775.107, values={1684130580=993775.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:25.027 INFO 11 --- [alarm-detect-2,,62fa32874c96] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:03:25.719 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:26.0, nextRun=2023-05-15 14:03:25.0, currentDate=Mon May 15 14:03:25 CST 2023 2023-05-15 14:03:25.719 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:02:26.0, nextRunTime=2023-05-15 14:03:25.0} 2023-05-15 14:03:25.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:03:25.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:25.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:25.949 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:25.951 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:25.954 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:25.961 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:25.964 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131871, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:25.966 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131871, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:03:25.962+08:00, endTime=null} 2023-05-15 14:03:25.984 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131871, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116273, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:25.962+08:00, sub_task_instance_id=131871}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:03:26.017 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:03:26.017 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:03:26.018 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:03:26.018 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:03:26.019 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:03:26.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c2c1a361ae9d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131871, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:26.713 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:26.714 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:26.934 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:26.997 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116273, status:SUCCESS 2023-05-15 14:03:26.998 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116273, status:SUCCESS 2023-05-15 14:03:27.352 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:27.875 INFO 11 --- [pool-ocp-schedules-13,,0c42faa66161] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:28.263 INFO 11 --- [pool-overview-container-3,,4275814bcb9d] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:03:28.263 INFO 11 --- [pool-overview-container-3,,4275814bcb9d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:28.510 INFO 11 --- [pool-scanner--15971296-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:29.714 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:29.716 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:30.200 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/DC20D776782F874E57959B272643EBAB/text?startTime=2023-05-15T13%3A33%3A21%2B08%3A00&endTime=2023-05-15T14%3A03%3A21%2B08%3A00, client=192.168.80.229, traceId=8ba488aa5299442a, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,DC20D776782F874E57959B272643EBAB,2023-05-15T13:33:21+08:00,2023-05-15T14:03:21+08:00,] 2023-05-15 14:03:30.212 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,9142b547ee3941d4,3f025cf6c128] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=9142b547ee3941d4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:30.216 INFO 11 --- [pool-ocp-schedules-12,,825e0fabc8a9] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:03:30.218 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,9142b547ee3941d4,3f025cf6c128] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=9142b547ee3941d4, duration=6 ms] 2023-05-15 14:03:30.232 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.233 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.233 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:30.234 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:30.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.238 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:30.239 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:30.263 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:30.271 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.272 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:30.273 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:30.289 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:30.289 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:30.290 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:03:30.437 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8ba488aa5299442a,6788d96a05a1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/DC20D776782F874E57959B272643EBAB/text client=192.168.80.229, traceId=8ba488aa5299442a, duration=242 ms] 2023-05-15 14:03:30.883 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:31.0, nextRun=2023-05-15 14:03:30.0, currentDate=Mon May 15 14:03:30 CST 2023 2023-05-15 14:03:30.883 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:02:31.0, nextRunTime=2023-05-15 14:03:30.0} 2023-05-15 14:03:30.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:03:30.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:30.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:30.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:30.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:01.0, nextRun=2023-05-15 14:03:30.0, currentDate=Mon May 15 14:03:30 CST 2023 2023-05-15 14:03:30.903 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:03:01.0, nextRunTime=2023-05-15 14:03:30.0} 2023-05-15 14:03:30.907 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:03:30.907 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:03:30.907 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:03:31.047 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131874, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:31.049 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131874, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:03:31.044+08:00, endTime=null} 2023-05-15 14:03:31.063 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131872, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:31.064 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131872, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:03:31.061+08:00, endTime=null} 2023-05-15 14:03:31.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,fd10865f6b68] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131874, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116275, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:31.044+08:00, sub_task_instance_id=131874}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:03:31.069 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,fd10865f6b68] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:03:31.077 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131873, operation:EXECUTE, state: RUNNING 2023-05-15 14:03:31.080 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131873, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:03:31.075+08:00, endTime=null} 2023-05-15 14:03:31.085 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131872, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116274, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:31.060+08:00, sub_task_instance_id=131872}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:03:31.088 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:03:31.089 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:03:31.089 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:03:31.089 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:03:31.089 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:03:31.089 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:03:31.099 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131873, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116274, task_operation=execute, latest_execution_start_time=2023-05-15T14:03:31.074+08:00, sub_task_instance_id=131873}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:03:31.100 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.100 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.100 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.101 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.101 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.101 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.102 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.103 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:03:31.104 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:03:31.105 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-1,fb27d19334114ed7,058895d8d28f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.106 INFO 11 --- [pool-3244136-thread-4,fb27d19334114ed7,8c69cd988aa0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:03:31.107 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:03:31.107 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:03:31.107 INFO 11 --- [pool-3244136-thread-3,fb27d19334114ed7,ebdafcdc9af3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:03:31.107 INFO 11 --- [pool-3244136-thread-2,fb27d19334114ed7,25867e1b5dcf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:03:31.109 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.110 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.110 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.110 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.111 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.111 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.112 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:03:31.115 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.115 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.115 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:03:31.117 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:03:31.117 INFO 11 --- [pool-3244136-thread-6,fb27d19334114ed7,6dae6ef6b1b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:03:31.121 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:03:31.121 INFO 11 --- [pool-3244136-thread-5,fb27d19334114ed7,03b08343a031] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:03:31.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,fd10865f6b68] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:03:31.122 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.122 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.122 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.123 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.127 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,fd10865f6b68] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131874, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:31.129 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.129 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.129 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:03:31.140 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.141 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.141 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.152 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.152 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.152 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.161 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.162 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.162 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.162 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.164 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.164 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.165 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.167 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.167 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.167 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:03:31.172 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.172 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.173 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.176 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.176 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.176 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.185 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.185 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.185 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:03:31.186 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.186 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.186 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.195 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:03:31.198 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.198 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.199 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.205 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.205 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.206 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:03:31.213 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.214 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.214 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.222 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.222 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.225 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:03:31.232 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:03:31.241 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:03:31.260 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.260 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.261 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.261 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.261 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.262 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.262 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.263 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.263 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.264 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.271 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.271 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.271 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.276 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.276 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.277 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.283 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.284 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.284 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.285 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.296 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.297 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.298 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.298 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.298 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.300 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.300 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.302 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.302 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.302 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.303 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.303 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.304 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.312 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.312 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.314 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.314 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.314 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.329 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.330 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.331 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.331 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.332 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.334 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.335 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.335 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.335 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.336 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.336 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.337 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.346 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.348 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.349 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.349 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.349 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.353 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.354 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.354 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.354 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.355 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.360 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.360 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.361 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.361 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:03:31.368 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.369 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.369 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.369 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.369 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:03:31.373 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:03:31.381 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.382 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.382 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.382 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.383 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.384 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.384 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.384 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:03:31.385 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:03:31.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:03:31.399 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.400 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.400 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.400 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.401 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.405 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:31.406 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.406 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.407 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.407 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.407 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.407 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.419 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:03:31.420 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.420 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.420 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:03:31.423 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.425 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:03:31.426 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:31.426 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.427 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:03:31.436 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.438 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:03:31.438 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.438 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.439 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:03:31.439 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.441 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:03:31.441 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:03:31.441 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:03:31.446 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,be9e8706ae7e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131873, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:31.448 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.450 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:03:31.450 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:31.450 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:31.451 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:03:31.460 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:03:31.462 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:03:31.462 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:03:31.462 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:03:31.466 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,5240b622454b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131872, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:03:31.801 INFO 11 --- [pool-ocp-schedules-2,,5f274e5e43e0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:31.801 INFO 11 --- [pool-ocp-schedules-2,,5f274e5e43e0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:31.996 INFO 11 --- [pool-ocp-schedules-12,,682d81717279] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:03:31.998 INFO 11 --- [pool-ocp-schedules-12,,682d81717279] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:03:32.019 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.019 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.019 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.019 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.021 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.021 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.021 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.021 INFO 11 --- [pool-ocp-schedules-15,,2b2fc66b8c37] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:03:32.022 INFO 11 --- [ForkJoinPool-66018-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:03:32.022 INFO 11 --- [pool-ocp-schedules-6,,f94682b6ffde] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:03:32.023 INFO 11 --- [pool-ocp-schedules-15,,2b2fc66b8c37] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:03:32.024 INFO 11 --- [pool-ocp-schedules-6,,f94682b6ffde] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:03:32.026 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.027 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.027 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.028 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.032 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.032 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.033 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.036 INFO 11 --- [ForkJoinPool-66018-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:03:32.044 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.044 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.044 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.044 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.052 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.052 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.052 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:03:32.055 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.055 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.055 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.056 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.060 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.060 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.060 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:03:32.084 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:32.085 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.085 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.085 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.085 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.086 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.086 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.086 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:03:32.087 WARN 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:03:32.087 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.087 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.087 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:32.088 INFO 11 --- [ForkJoinPool-66020-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:03:32.088 INFO 11 --- [ForkJoinPool-66020-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:03:32.088 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:03:32.090 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116274, status:SUCCESS 2023-05-15 14:03:32.091 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116274, status:SUCCESS 2023-05-15 14:03:32.098 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116275, status:SUCCESS 2023-05-15 14:03:32.099 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.099 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:03:32.099 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116275, status:SUCCESS 2023-05-15 14:03:32.111 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.119 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:32.120 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.120 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.121 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.121 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.122 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:03:32.122 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.123 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.129 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:32.130 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.131 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.131 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.132 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.132 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:03:32.132 INFO 11 --- [ForkJoinPool-66020-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:03:32.136 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.136 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.137 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:03:32.141 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.141 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.142 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:03:32.145 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:03:32.164 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.164 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:03:32.184 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.184 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:03:32.203 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.203 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:03:32.226 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.226 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:03:32.246 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.246 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:03:32.265 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:03:32.284 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:32.285 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.286 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.286 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.287 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.291 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:32.291 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:32.292 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:32.313 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:03:32.313 INFO 11 --- [ForkJoinPool-66020-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:03:32.716 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:32.717 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:34.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:34.829118+08:00, lastProcessTime=2023-05-15T14:03:24.827053+08:00 2023-05-15 14:03:34.863 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/0AC10BF752225B69BE0126BC434A8A14/text?startTime=2023-05-15T13%3A33%3A21%2B08%3A00&endTime=2023-05-15T14%3A03%3A21%2B08%3A00, client=192.168.80.229, traceId=73caaf559b71497e, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,0AC10BF752225B69BE0126BC434A8A14,2023-05-15T13:33:21+08:00,2023-05-15T14:03:21+08:00,] 2023-05-15 14:03:34.865 INFO 11 --- [alarm-detect-5,,f7111306ec6a] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130594 ~ 1684130604 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130600=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:34.879 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:34.829118+08:00 2023-05-15 14:03:34.885 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:34.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:34.891 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.891 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:34.892 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:03:34.902 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:03:34.903 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.904 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.904 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:34.904 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:03:34.910 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:03:34.910 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:03:34.911 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:03:35.098 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,73caaf559b71497e,beb6d72a46ee] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/0AC10BF752225B69BE0126BC434A8A14/text client=192.168.80.229, traceId=73caaf559b71497e, duration=236 ms] 2023-05-15 14:03:35.717 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:35.718 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:35.951 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:35.954 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:35.956 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:35.962 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:36.903 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,73cb63e6a5094b2c,b28f3f36faa5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=73cb63e6a5094b2c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:36.909 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,73cb63e6a5094b2c,b28f3f36faa5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=73cb63e6a5094b2c, duration=7 ms] 2023-05-15 14:03:36.937 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:37.897 INFO 11 --- [pool-ocp-schedules-15,,610c7bbee9ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:38.261 INFO 11 --- [pool-overview-container-4,,8f87bbdea730] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:38.509 INFO 11 --- [pool-scanner--15971381-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:38.718 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:38.721 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:41.721 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:41.722 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:42.355 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:44.722 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:44.723 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:44.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:44.830265+08:00, lastProcessTime=2023-05-15T14:03:34.829118+08:00 2023-05-15 14:03:44.845 INFO 11 --- [alarm-detect-5,,3d4255485397] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130604 ~ 1684130614 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130610=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:44.878 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:44.830265+08:00 2023-05-15 14:03:45.400 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,6c23848f4f1b4c82,680fd0511fc6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=6c23848f4f1b4c82, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:03:45.402 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,6c23848f4f1b4c82,680fd0511fc6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=6c23848f4f1b4c82, duration=2 ms] 2023-05-15 14:03:45.474 INFO 11 --- [pool-ocp-schedules-13,,c768f477ed41] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:03:45.954 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:45.956 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:45.959 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:45.964 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:46.940 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:47.724 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:47.725 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:47.918 INFO 11 --- [pool-ocp-schedules-6,,a0e431984fd6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:48.261 INFO 11 --- [pool-overview-container-8,,c93256cfc638] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:48.509 INFO 11 --- [pool-scanner--15971481-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:49.526 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,71541870794a4fcb,a027e8d6b4e1] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=71541870794a4fcb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:49.531 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,71541870794a4fcb,a027e8d6b4e1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=71541870794a4fcb, duration=4 ms] 2023-05-15 14:03:49.792 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,8dfcaf4c2e8e4451,54dfb7663c15] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8dfcaf4c2e8e4451, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:49.798 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,8dfcaf4c2e8e4451,54dfb7663c15] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8dfcaf4c2e8e4451, duration=7 ms] 2023-05-15 14:03:50.725 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:50.726 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:51.805 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:51.807 INFO 11 --- [pool-ocp-schedules-12,,92d4b41d156d] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:51.807 INFO 11 --- [pool-ocp-schedules-12,,92d4b41d156d] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:03:51.807 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:51.809 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,09cdb502859a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2700,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40033} 2023-05-15 14:03:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,09cdb502859a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:03:53.726 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:53.727 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:03:54.837 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:03:54.837224+08:00, lastProcessTime=2023-05-15T14:03:44.830265+08:00 2023-05-15 14:03:54.882 INFO 11 --- [alarm-detect-7,,128239f9b88e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130614 ~ 1684130624 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130620=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:03:54.933 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:03:54.837224+08:00 2023-05-15 14:03:54.965 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,f59c54e57ff44415,b70e599edef4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=f59c54e57ff44415, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:03:54.969 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,f59c54e57ff44415,b70e599edef4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=f59c54e57ff44415, duration=5 ms] 2023-05-15 14:03:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,d233ba9f52b1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:03:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,ce022222c625] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:03:55.957 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:55.960 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:55.962 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:55.966 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:56.727 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:56.728 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:03:56.944 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:57.357 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:03:57.939 INFO 11 --- [pool-ocp-schedules-2,,aa1b9748a991] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:03:58.262 INFO 11 --- [pool-overview-container-5,,d302f863c438] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:03:58.510 INFO 11 --- [pool-scanner--15971575-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:03:59.728 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:03:59.729 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:04:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:04:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:04:00.001 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130640 2023-05-15 14:04:00.051 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:04:00.775 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:02:01.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.775 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:02:01.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:04:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.783 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:00.797 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:01.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.797 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:03:01.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.802 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:04:00.802 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.802 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:00.808 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:01.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.808 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:03:01.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.820 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:04:00.820 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.820 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:00.829 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:01.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.829 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:03:01.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:04:00.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:00.848 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:31.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.849 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:03:31.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.852 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:04:00.852 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.852 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:00.858 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:01.0, nextRun=2023-05-15 14:04:00.0, currentDate=Mon May 15 14:04:00 CST 2023 2023-05-15 14:04:00.858 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:03:01.0, nextRunTime=2023-05-15 14:04:00.0} 2023-05-15 14:04:00.861 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:04:00.861 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:00.861 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:01.501 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131875, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.504 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131875, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:04:01.499+08:00, endTime=null} 2023-05-15 14:04:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,3ef24f98b04b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130700, interval=60 2023-05-15 14:04:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,f35866a609d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130700, interval=60 2023-05-15 14:04:01.515 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131878, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.517 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131878, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:04:01.513+08:00, endTime=null} 2023-05-15 14:04:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,3ef24f98b04b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:04:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,3ef24f98b04b] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130641, count=8, elapsed=1682446510882 2023-05-15 14:04:01.526 INFO 11 --- [pool-prometheus-minute-schedule-2,,f35866a609d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:04:01.526 INFO 11 --- [pool-prometheus-minute-schedule-2,,f35866a609d6] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130641, count=6, elapsed=1682446510885 2023-05-15 14:04:01.527 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,45845c34bf39] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131875, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116276, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.498+08:00, sub_task_instance_id=131875}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.530 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131879, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.532 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131879, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:04:01.527+08:00, endTime=null} 2023-05-15 14:04:01.540 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d02a607e008a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131878, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116278, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.513+08:00, sub_task_instance_id=131878}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.547 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131876, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.550 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131876, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:04:01.544+08:00, endTime=null} 2023-05-15 14:04:01.552 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,65d92fad6580] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131879, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116279, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.527+08:00, sub_task_instance_id=131879}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.553 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,45845c34bf39] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131875, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.564 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131877, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.566 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131877, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:04:01.561+08:00, endTime=null} 2023-05-15 14:04:01.569 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d02a607e008a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:04:01.570 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d02a607e008a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:04:01.571 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2302431fe2bd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131876, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116276, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.544+08:00, sub_task_instance_id=131876}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.575 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d02a607e008a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131878, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.583 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131880, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.583 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,65d92fad6580] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:04:01.583 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,65d92fad6580] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:04:01.584 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2302431fe2bd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131876, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.584 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131880, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:04:01.574+08:00, endTime=null} 2023-05-15 14:04:01.585 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,65d92fad6580] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:04:01.589 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2ff1222968b0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131877, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116277, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.559+08:00, sub_task_instance_id=131877}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.589 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,65d92fad6580] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131879, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.592 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131881, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:01.593 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131881, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:04:01.590+08:00, endTime=null} 2023-05-15 14:04:01.594 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:04:01.594 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:04:01.594 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:04:01.594 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:04:01.594 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:04:01.600 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:04:01.600 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:04:01.602 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,2ff1222968b0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131877, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.603 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:04:01.604 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c81b50fa5baa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131880, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116280, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.574+08:00, sub_task_instance_id=131880}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.604 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c81b50fa5baa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:04:01.607 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:04:01.609 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:04:01.611 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116276, status:SUCCESS 2023-05-15 14:04:01.612 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116276, status:SUCCESS 2023-05-15 14:04:01.616 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c3ead08f5c81] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131881, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116281, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:01.590+08:00, sub_task_instance_id=131881}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:01.620 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116277, status:SUCCESS 2023-05-15 14:04:01.626 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116277, status:SUCCESS 2023-05-15 14:04:01.629 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:04:01.630 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:04:01.630 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:04:01.638 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c3ead08f5c81] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131881, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:01.639 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:04:01.640 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:04:01.640 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116278, status:SUCCESS 2023-05-15 14:04:01.641 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116278, status:SUCCESS 2023-05-15 14:04:01.644 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:04:01.649 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.649 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.650 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.650 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.650 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.651 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.651 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.651 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.651 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.651 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.652 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:01.653 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.654 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:04:01.654 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-3,fb27d19334114ed7,69bb6270e27d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.655 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116279, status:SUCCESS 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-4,fb27d19334114ed7,b83633d0cd86] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-2,fb27d19334114ed7,9c9c8c0e7d05] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:04:01.655 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:04:01.656 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.656 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.656 INFO 11 --- [pool-3244290-thread-1,fb27d19334114ed7,7d549c6b77d1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:04:01.656 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.656 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116279, status:SUCCESS 2023-05-15 14:04:01.656 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:01.661 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.661 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.661 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:01.661 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:01.662 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:04:01.662 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:04:01.665 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116281, status:SUCCESS 2023-05-15 14:04:01.665 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116281, status:SUCCESS 2023-05-15 14:04:01.667 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:04:01.667 INFO 11 --- [pool-3244290-thread-6,fb27d19334114ed7,30b1594f0534] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:04:01.667 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:04:01.667 INFO 11 --- [pool-3244290-thread-5,fb27d19334114ed7,5c1c0f91e3d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:04:01.668 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c81b50fa5baa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:04:01.671 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c81b50fa5baa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131880, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:02.697 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116280, status:SUCCESS 2023-05-15 14:04:02.698 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116280, status:SUCCESS 2023-05-15 14:04:02.729 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:02.730 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:04.110 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,5dc6dd55373c4c29,c1a6b08414e5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=5dc6dd55373c4c29, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:04.116 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,5dc6dd55373c4c29,c1a6b08414e5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=5dc6dd55373c4c29, duration=6 ms] 2023-05-15 14:04:04.714 INFO 11 --- [pool-ocp-schedules-17,,b9a2d30a6d92] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:04:04.723 INFO 11 --- [pool-ocp-schedules-17,,b9a2d30a6d92] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:04:04.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:04.835 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2023-05-15 14:04:04.840 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2023-05-15 14:04:04.844 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:04.843763+08:00, lastProcessTime=2023-05-15T14:03:54.837224+08:00 2023-05-15 14:04:04.863 INFO 11 --- [alarm-detect-2,,af7584aa0e6b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130624 ~ 1684130634 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130630=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:04.909 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:04.843763+08:00 2023-05-15 14:04:05.150 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,84e22cbf52584b33,0bae1b43eb70] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=84e22cbf52584b33, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:05.156 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,84e22cbf52584b33,0bae1b43eb70] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=84e22cbf52584b33, duration=6 ms] 2023-05-15 14:04:05.730 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:05.732 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:05.960 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:05.963 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:05.965 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:05.968 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:05.995 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:05.0, nextRun=2023-05-15 14:04:05.0, currentDate=Mon May 15 14:04:05 CST 2023 2023-05-15 14:04:05.996 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:03:05.0, nextRunTime=2023-05-15 14:04:05.0} 2023-05-15 14:04:06.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:04:06.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:06.000 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:06.681 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131882, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:06.683 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131882, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:04:06.678+08:00, endTime=null} 2023-05-15 14:04:06.705 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a94a09fbf6fa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131882, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116282, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:06.678+08:00, sub_task_instance_id=131882}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:06.731 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,d9e5b20e5f3b] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:04:06.732 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,576475359c2f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:04:06.733 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,5d2196bf4967] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:04:06.769 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,576475359c2f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:04:06.772 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,d9e5b20e5f3b] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:04:06.775 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,5d2196bf4967] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:04:06.784 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a94a09fbf6fa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131882, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:06.947 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:07.805 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116282, status:SUCCESS 2023-05-15 14:04:07.806 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116282, status:SUCCESS 2023-05-15 14:04:07.960 INFO 11 --- [pool-ocp-schedules-17,,47162a64bf4b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:08.263 INFO 11 --- [pool-overview-container-3,,254357399188] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:04:08.263 INFO 11 --- [pool-overview-container-3,,254357399188] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:08.510 INFO 11 --- [pool-scanner--15971668-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:04:08.732 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:08.733 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:10.236 INFO 11 --- [pool-ocp-schedules-7,,0746e1eb84a8] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:04:11.733 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:11.734 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:11.813 INFO 11 --- [pool-ocp-schedules-17,,9202cc0d6d1c] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:11.813 INFO 11 --- [pool-ocp-schedules-17,,9202cc0d6d1c] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:12.360 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:14.359 INFO 11 --- [pool-ocp-schedules-4,,7cd1d2c6391c] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:04:14.360 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:04:14.359 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:04:14.360 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:04:14.360 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:04:14.360 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:04:14.365 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:04:14.366 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:04:14.367 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:04:14.367 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:04:14.368 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:04:14.383 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130580, nodeCount=556, affectRows=556 2023-05-15 14:04:14.386 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130580, nodeCount=1000, affectRows=1000 2023-05-15 14:04:14.387 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130580, nodeCount=556, validCount=556 2023-05-15 14:04:14.388 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130580, nodeCount=572, affectRows=572 2023-05-15 14:04:14.390 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130580, nodeCount=1000, validCount=1000 2023-05-15 14:04:14.390 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130580, nodeCount=572, validCount=572 2023-05-15 14:04:14.392 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130580, nodeCount=572, affectRows=572 2023-05-15 14:04:14.393 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:04:14.394 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:04:14.395 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130580, nodeCount=572, validCount=572 2023-05-15 14:04:14.395 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:04:14.396 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:04:14.396 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:04:14.396124538+08:00 2023-05-15 14:04:14.396 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:04:14.395726342+08:00 2023-05-15 14:04:14.396 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:04:14.396+08:00 2023-05-15 14:04:14.396 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:04:14.396+08:00 2023-05-15 14:04:14.397 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:04:14.397 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:04:14.397275462+08:00 2023-05-15 14:04:14.398 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:04:14.397+08:00 2023-05-15 14:04:14.398 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:04:14.398086813+08:00 2023-05-15 14:04:14.399 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:04:14.398+08:00 2023-05-15 14:04:14.399 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:04:14.399036554+08:00 2023-05-15 14:04:14.399 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:04:14.399+08:00 2023-05-15 14:04:14.400 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:04:14.396+08:00 and status AVAILABLE 2023-05-15 14:04:14.400 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:04:14.397+08:00 and status AVAILABLE 2023-05-15 14:04:14.401 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:04:14.396+08:00 and status AVAILABLE 2023-05-15 14:04:14.402 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:04:14.399+08:00 and status AVAILABLE 2023-05-15 14:04:14.403 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:04:14.398+08:00 and status AVAILABLE 2023-05-15 14:04:14.407 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:04:14.407 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:04:14.408 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:04:14.408 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:04:14.410 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:04:14.419 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130654 0 0 , error= 2023-05-15 14:04:14.419 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130654 0 0 , error= 2023-05-15 14:04:14.427 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:04:14.419+08:00 2023-05-15 14:04:14.427 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130654 0 0 , error= 2023-05-15 14:04:14.436 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:04:14.420+08:00 2023-05-15 14:04:14.436 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:04:14.436+08:00 2023-05-15 14:04:14.440 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130654 0 0 , error= 2023-05-15 14:04:14.461 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130654 0 0 , error= 2023-05-15 14:04:14.462 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:04:14.462+08:00 2023-05-15 14:04:14.462 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:04:14.462+08:00 2023-05-15 14:04:14.466 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:04:14.436+08:00 and status AVAILABLE 2023-05-15 14:04:14.466 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:04:14.462+08:00 and status AVAILABLE 2023-05-15 14:04:14.466 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:04:14.419+08:00 and status AVAILABLE 2023-05-15 14:04:14.467 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:04:14.420+08:00 and status AVAILABLE 2023-05-15 14:04:14.468 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:04:14.462+08:00 and status AVAILABLE 2023-05-15 14:04:14.469 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:04:14.469 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:04:14.469 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:04:14.469 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:04:14.470 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:04:14.479 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:04:14.481 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:04:14.481 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:04:14.482 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:04:14.482 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:04:14.510 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:04:14.510 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:04:14.511 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:04:14.512 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:04:14.513 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:04:14.556 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:04:14.557 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:04:14.567 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:04:14.569 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:04:14.572 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:04:14.573 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:04:14.583 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:04:14.583 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:04:14.594 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:04:14.595 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:04:14.595 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:04:14.602 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:04:14.604 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:04:14.607 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:04:14.625 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:04:14.637 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:04:14.652 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:04:14.657 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:04:14.658 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:04:14.660 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:04:14.668 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:04:14.670 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:04:14.673 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:04:14.679 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:04:14.686 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:04:14.687 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:04:14.688 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:04:14.700 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:04:14.711 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:04:14.731 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:04:14.732 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:04:14.733 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:04:14.734 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:14.735 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:14.741 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:04:14.744 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:04:14.745 INFO 11 --- [pool-Check host status30,,4fbd26ab33f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:04:14.746 INFO 11 --- [pool-Check host status13,,b844636cb119] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:04:14.755 INFO 11 --- [pool-Check host status8,,634e8381fefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:04:14.756 INFO 11 --- [pool-Check host status14,,e9059c945d1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:04:14.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:14.806 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:04:14.818 INFO 11 --- [pool-Check host status1,,d9f2a8805ef7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:04:14.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:14.827582+08:00, lastProcessTime=2023-05-15T14:04:04.843763+08:00 2023-05-15 14:04:14.839 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:04:14.839 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:04:14.839 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:04:14.839 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:04:14.839 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:04:14.848 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:04:14.850 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:04:14.851 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:04:14.851 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:04:14.851 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:04:14.863 INFO 11 --- [alarm-detect-12,,8a6f4a8f68a9] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130634 ~ 1684130644 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130640=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:14.879 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:04:14.879 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:04:14.880 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:04:14.881 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:04:14.881 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:04:14.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:14.827582+08:00 2023-05-15 14:04:14.897 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:04:14.897 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:04:14.897 INFO 11 --- [pool-Check host status17,,f5d008c9cdaf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:04:14.899 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status19,,f87c5e76f1c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status2,,d679246935de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:04:14.900 INFO 11 --- [pool-Check host status28,,18eff8157101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:04:14.901 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:04:14.901 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:04:14.901 INFO 11 --- [pool-Check host status3,,72195b5f1adf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:04:15.963 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:15.966 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:15.967 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:15.970 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:16.950 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:17.736 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:17.737 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:17.982 INFO 11 --- [pool-ocp-schedules-15,,73ad0a37bd50] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:18.262 INFO 11 --- [pool-overview-container-7,,1a1c71f93912] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:18.511 INFO 11 --- [pool-scanner--15971768-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:19.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,50ffcce7eba84381,b80699a0ccb2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=50ffcce7eba84381, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:04:19.436 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,50ffcce7eba84381,b80699a0ccb2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=50ffcce7eba84381, duration=2 ms] 2023-05-15 14:04:20.444 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:21.0, nextRun=2023-05-15 14:04:20.0, currentDate=Mon May 15 14:04:20 CST 2023 2023-05-15 14:04:20.444 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:03:21.0, nextRunTime=2023-05-15 14:04:20.0} 2023-05-15 14:04:20.448 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:04:20.448 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:20.448 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:20.737 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:20.738 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:20.888 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131883, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:20.889 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131883, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:04:20.886+08:00, endTime=null} 2023-05-15 14:04:20.910 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,373ee17355ff] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131883, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116283, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:20.885+08:00, sub_task_instance_id=131883}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:20.912 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,373ee17355ff] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:04:20.912 INFO 11 --- [ForkJoinPool-66023-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:04:20.913 INFO 11 --- [ForkJoinPool-66023-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:04:20.918 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,373ee17355ff] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131883, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:21.062 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116283, status:SUCCESS 2023-05-15 14:04:21.063 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116283, status:SUCCESS 2023-05-15 14:04:21.273 INFO 11 --- [pool-scanner--15971769-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:21.294 INFO 11 --- [pool-scanner--15971770-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:21.318 INFO 11 --- [pool-scanner--15971771-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:21.341 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.346 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.392 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.396 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.416 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.421 INFO 11 --- [pool-ocp-schedules-7,,e4763bcd0a49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:21.807 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:21.810 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:21.811 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:23.691 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,d433bcb235534509,c956e62d543f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=d433bcb235534509, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:23.697 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,d433bcb235534509,c956e62d543f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=d433bcb235534509, duration=6 ms] 2023-05-15 14:04:23.738 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:23.739 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:24.282 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,86baf765ce694bc6,a285a5c19eba] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=86baf765ce694bc6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:24.287 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,86baf765ce694bc6,a285a5c19eba] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=86baf765ce694bc6, duration=6 ms] 2023-05-15 14:04:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:24.830222+08:00, lastProcessTime=2023-05-15T14:04:14.827582+08:00 2023-05-15 14:04:24.869 INFO 11 --- [alarm-detect-7,,505f759535b3] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130644 ~ 1684130654 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130650=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:24.912 INFO 11 --- [alarm-detect-4,,d96830f5177c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130594 ~ 1684130654 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389037.988, values={1684130640=389037.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:04:24.912 INFO 11 --- [alarm-detect-4,,d96830f5177c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130594 ~ 1684130654 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993835.358, values={1684130640=993835.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:24.913 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:24.830222+08:00 2023-05-15 14:04:25.590 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:26.0, nextRun=2023-05-15 14:04:25.0, currentDate=Mon May 15 14:04:25 CST 2023 2023-05-15 14:04:25.590 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:03:26.0, nextRunTime=2023-05-15 14:04:25.0} 2023-05-15 14:04:25.602 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:04:25.602 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:25.602 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:25.966 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:25.968 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:25.970 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:25.972 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:25.975 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131884, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:25.977 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131884, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:04:25.971+08:00, endTime=null} 2023-05-15 14:04:26.000 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131884, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116284, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:25.971+08:00, sub_task_instance_id=131884}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:04:26.043 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:04:26.043 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:04:26.044 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:04:26.044 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:04:26.045 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:04:26.050 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1e1c6a30bc79] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131884, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:26.166 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116284, status:SUCCESS 2023-05-15 14:04:26.167 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116284, status:SUCCESS 2023-05-15 14:04:26.739 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:26.740 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:26.953 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:27.362 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:28.011 INFO 11 --- [pool-ocp-schedules-19,,4ed9ce006419] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:28.262 INFO 11 --- [pool-overview-container-1,,3ceb68d17a25] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:28.510 INFO 11 --- [pool-scanner--15971869-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:29.740 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:29.741 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:30.743 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:03:31.0, nextRun=2023-05-15 14:04:30.0, currentDate=Mon May 15 14:04:30 CST 2023 2023-05-15 14:04:30.743 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:03:31.0, nextRunTime=2023-05-15 14:04:30.0} 2023-05-15 14:04:30.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:04:30.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:30.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:30.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:30.758 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:04:30.0, currentDate=Mon May 15 14:04:30 CST 2023 2023-05-15 14:04:30.758 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:04:30.0} 2023-05-15 14:04:30.761 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:04:30.761 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:04:30.761 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:04:31.067 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131886, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:31.069 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131886, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:04:31.064+08:00, endTime=null} 2023-05-15 14:04:31.084 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131887, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:31.087 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131887, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:04:31.081+08:00, endTime=null} 2023-05-15 14:04:31.094 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131886, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116285, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:31.064+08:00, sub_task_instance_id=131886}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:04:31.096 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131885, operation:EXECUTE, state: RUNNING 2023-05-15 14:04:31.098 INFO 11 --- [pool-subtask-coordinator3,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131885, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:04:31.094+08:00, endTime=null} 2023-05-15 14:04:31.104 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ff1603f49db2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131887, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116286, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:31.080+08:00, sub_task_instance_id=131887}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:04:31.104 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ff1603f49db2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:04:31.114 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131885, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116285, task_operation=execute, latest_execution_start_time=2023-05-15T14:04:31.094+08:00, sub_task_instance_id=131885}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:04:31.122 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:04:31.136 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.136 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.136 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.136 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.136 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.137 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.138 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:04:31.139 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:04:31.141 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-1,fb27d19334114ed7,9d25033d617c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-2,fb27d19334114ed7,ba3da9d2967e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.142 INFO 11 --- [pool-3244448-thread-4,fb27d19334114ed7,7cf91f83c1d5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:04:31.143 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:04:31.143 INFO 11 --- [pool-3244448-thread-3,fb27d19334114ed7,ad9d35bbf2db] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:04:31.144 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.144 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.144 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.145 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.148 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.148 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.149 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:04:31.149 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.149 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.150 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:04:31.152 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:04:31.152 INFO 11 --- [pool-3244448-thread-5,fb27d19334114ed7,fa27c9d1b1c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:04:31.153 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.154 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.154 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.154 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.154 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:04:31.154 INFO 11 --- [pool-3244448-thread-6,fb27d19334114ed7,cc0d03d4a961] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:04:31.155 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.155 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.155 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ff1603f49db2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:04:31.155 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:04:31.160 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ff1603f49db2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131887, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:31.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.161 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.162 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.162 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.162 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.166 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.166 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.167 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:04:31.170 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.170 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.170 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.172 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.172 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.173 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.191 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.191 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.192 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.198 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.198 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.199 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.203 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.203 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.204 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.217 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.217 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.217 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.218 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.218 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.218 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.228 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:04:31.228 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.228 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.229 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:04:31.238 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.238 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.239 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:04:31.248 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.248 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.249 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:04:31.253 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.254 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.255 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.255 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.256 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.261 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.261 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.262 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.268 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.268 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.268 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:04:31.278 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:04:31.282 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.283 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.284 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.284 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.284 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.288 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.288 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.289 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.298 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.299 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.300 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.307 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116286, status:SUCCESS 2023-05-15 14:04:31.308 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116286, status:SUCCESS 2023-05-15 14:04:31.311 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.311 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.312 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.312 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.312 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.313 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.315 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.315 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.316 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.316 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.316 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.320 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.320 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.321 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.331 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.331 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.332 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.333 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.335 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.335 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.336 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.336 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.336 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.342 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.342 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:04:31.344 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.345 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.345 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:04:31.345 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.345 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.346 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.347 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.371 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.372 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.372 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.372 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.374 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.374 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.374 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.382 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.383 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.383 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.383 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.384 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.386 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.386 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.387 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.388 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.390 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.390 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.390 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.401 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.403 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.403 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.403 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.403 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:04:31.403 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.404 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.404 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:31.404 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.405 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.405 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.405 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:04:31.415 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.416 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.416 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.416 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.416 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.416 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.417 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:04:31.418 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:04:31.418 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:04:31.418 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:04:31.418 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:04:31.420 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:04:31.423 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1ddb243d4605] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131886, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:31.440 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:31.440 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.441 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.441 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.441 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.444 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:04:31.456 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.458 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:04:31.458 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.458 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.459 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:04:31.475 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.477 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:04:31.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.478 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:04:31.487 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.489 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:04:31.489 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:31.490 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:31.490 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:04:31.499 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:04:31.500 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:04:31.501 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:04:31.501 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:04:31.505 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,c865b9a83f42] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131885, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:04:31.818 INFO 11 --- [pool-ocp-schedules-4,,63d428f89a24] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:31.818 INFO 11 --- [pool-ocp-schedules-4,,63d428f89a24] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:32.003 INFO 11 --- [pool-ocp-schedules-10,,9aae6352511d] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:04:32.005 INFO 11 --- [pool-ocp-schedules-10,,9aae6352511d] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:04:32.027 INFO 11 --- [pool-ocp-schedules-6,,275c8f883542] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:04:32.028 INFO 11 --- [pool-ocp-schedules-6,,275c8f883542] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:04:32.029 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.029 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.029 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.029 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.030 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.030 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.030 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.031 INFO 11 --- [pool-ocp-schedules-13,,c07f332d95b1] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:04:32.031 INFO 11 --- [ForkJoinPool-66024-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:04:32.032 INFO 11 --- [pool-ocp-schedules-13,,c07f332d95b1] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:04:32.035 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.035 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.035 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.036 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.044 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.044 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.044 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.048 INFO 11 --- [ForkJoinPool-66024-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:04:32.049 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.050 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.050 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.050 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.051 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.051 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.051 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:04:32.061 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.061 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.061 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.062 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.066 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.066 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.066 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:04:32.095 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:32.095 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.095 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.096 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.096 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.097 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.097 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.097 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:04:32.098 WARN 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:04:32.099 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.099 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.099 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:04:32.101 INFO 11 --- [ForkJoinPool-66026-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:04:32.101 INFO 11 --- [ForkJoinPool-66026-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:04:32.101 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:04:32.111 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.111 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:04:32.121 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.133 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:32.133 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.133 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.134 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.134 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.135 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:04:32.135 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.135 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.140 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:32.141 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.142 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.142 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.143 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.145 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:04:32.145 INFO 11 --- [ForkJoinPool-66026-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:04:32.147 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.147 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.148 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:04:32.151 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.152 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.152 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:04:32.155 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:04:32.173 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.174 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:04:32.190 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.191 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:04:32.202 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.203 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:04:32.215 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.216 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:04:32.226 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.226 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:04:32.237 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:04:32.249 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:04:32.250 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.251 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.251 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.251 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.256 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:04:32.256 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:04:32.256 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:04:32.273 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:04:32.273 INFO 11 --- [ForkJoinPool-66026-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:04:32.335 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116285, status:SUCCESS 2023-05-15 14:04:32.336 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116285, status:SUCCESS 2023-05-15 14:04:32.741 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:32.743 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:34.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:34.830054+08:00, lastProcessTime=2023-05-15T14:04:24.830222+08:00 2023-05-15 14:04:34.876 INFO 11 --- [alarm-detect-12,,1cbdffcbfbea] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130654 ~ 1684130664 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130660=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:34.884 INFO 11 --- [alarm-detect-8,,9fa353ebdef0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130604 ~ 1684130664 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389037.938, values={1684130640=389037.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:04:34.885 INFO 11 --- [alarm-detect-8,,9fa353ebdef0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130604 ~ 1684130664 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993835.107, values={1684130640=993835.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:34.912 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:34.830054+08:00 2023-05-15 14:04:35.743 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:35.744 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:35.969 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:35.971 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:35.973 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:35.975 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:36.956 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:38.009 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,1a5b38ec3e1740bf,af5a512651ca] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=1a5b38ec3e1740bf, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:38.014 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,1a5b38ec3e1740bf,af5a512651ca] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=1a5b38ec3e1740bf, duration=6 ms] 2023-05-15 14:04:38.031 INFO 11 --- [pool-ocp-schedules-16,,22d73974a1ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:38.262 INFO 11 --- [pool-overview-container-6,,cd89bf33375d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:38.511 INFO 11 --- [pool-scanner--15971980-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:38.744 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:38.745 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:41.745 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:41.747 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:42.336 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9fe158fe258f4965,b1483a53bdd5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=9fe158fe258f4965, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:42.342 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9fe158fe258f4965,b1483a53bdd5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=9fe158fe258f4965, duration=7 ms] 2023-05-15 14:04:42.365 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:44.747 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:44.748 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:44.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:44.833929+08:00, lastProcessTime=2023-05-15T14:04:34.830054+08:00 2023-05-15 14:04:44.860 INFO 11 --- [alarm-detect-7,,462acaf3a29b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130664 ~ 1684130674 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130670=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:44.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:44.833929+08:00 2023-05-15 14:04:45.482 INFO 11 --- [pool-ocp-schedules-20,,078bfafc093a] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:04:45.971 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:45.974 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:45.976 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:45.978 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:46.253 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,01406e7339f446d0,346bc9589ec5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=01406e7339f446d0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:46.258 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,01406e7339f446d0,346bc9589ec5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=01406e7339f446d0, duration=6 ms] 2023-05-15 14:04:46.869 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866} 2023-05-15 14:04:46.903 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.904 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.905 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.906 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.907 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.908 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.916 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.917 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.922 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.922 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.923 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:00:46.802, end=2023-05-15T14:02:46.866},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:04:46.923 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:04:46.923 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:04:46.959 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:47.748 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:47.749 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:48.051 INFO 11 --- [pool-ocp-schedules-14,,fd49462dab4d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:48.263 INFO 11 --- [pool-overview-container-2,,a357d4e26ee6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:48.512 INFO 11 --- [pool-scanner--15972080-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:50.750 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:50.751 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:51.810 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:51.812 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:51.814 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:51.823 INFO 11 --- [pool-ocp-schedules-3,,b1ea32f15c99] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:51.823 INFO 11 --- [pool-ocp-schedules-3,,b1ea32f15c99] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:04:53.461 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,cb798f3ae24a4dc5,ad87d8c37bdc] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=cb798f3ae24a4dc5, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:04:53.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,cb798f3ae24a4dc5,ad87d8c37bdc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=cb798f3ae24a4dc5, duration=2 ms] 2023-05-15 14:04:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,3c2beae3427f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2700,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40046} 2023-05-15 14:04:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,3c2beae3427f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:04:53.619 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.654 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.660 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.664 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.670 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.670 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.677 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.683 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 9 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.704 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 26 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.747 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 132 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.752 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:53.753 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:53.780 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.783 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.785 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.787 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.789 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.789 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.791 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.793 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.795 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.796 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.796 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.798 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.799 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.801 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.802 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.803 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.804 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.805 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.806 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.807 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:58:53.39, end=2023-05-15T14:03:53.617} 2023-05-15 14:04:53.807 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:04:53.807 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:04:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:04:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:04:54.829981+08:00, lastProcessTime=2023-05-15T14:04:44.833929+08:00 2023-05-15 14:04:54.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:04:54.829981+08:00 2023-05-15 14:04:54.934 INFO 11 --- [alarm-detect-10,,e43622ce73e2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130674 ~ 1684130684 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130680=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:04:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,39fc336edfd8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:04:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,ca8a1d8e850b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:04:55.974 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:55.976 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:55.978 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:55.980 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:56.753 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:56.754 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:04:56.962 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:57.367 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:04:57.589 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,7af2b558b6d24b9b,ad819734aa25] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=7af2b558b6d24b9b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:04:57.596 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,7af2b558b6d24b9b,ad819734aa25] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=7af2b558b6d24b9b, duration=8 ms] 2023-05-15 14:04:58.095 INFO 11 --- [pool-ocp-schedules-6,,3324cf997fa8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:04:58.263 INFO 11 --- [pool-overview-container-4,,2aabd9227450] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:04:58.515 INFO 11 --- [pool-scanner--15972174-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:04:59.754 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:04:59.755 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:05:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:05:00.000 INFO 11 --- [pool-ocp-schedules-6,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130700 2023-05-15 14:05:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:05:00.047 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:05:00.624 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.624 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.627 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-05-15 14:05:00.627 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.627 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.633 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.633 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.660 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:05:00.660 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.660 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.669 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:05:00.669 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.669 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.675 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.675 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:05:00.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:05:00.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.714 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:31.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.714 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:04:31.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.717 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:05:00.717 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.717 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.727 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-05-15 14:05:00.727 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.727 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.734 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.734 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.771 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:05:00.772 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.772 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:00.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:05:00.0, currentDate=Mon May 15 14:05:00 CST 2023 2023-05-15 14:05:00.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:05:00.0} 2023-05-15 14:05:00.781 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:05:00.782 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:00.782 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,6311e074e749] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130760, interval=60 2023-05-15 14:05:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,cc690ac52b6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130760, interval=60 2023-05-15 14:05:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,cc690ac52b6c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:05:01.522 INFO 11 --- [pool-prometheus-minute-schedule-1,,6311e074e749] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:05:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,cc690ac52b6c] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130701, count=6, elapsed=1682446570821 2023-05-15 14:05:01.522 INFO 11 --- [pool-prometheus-minute-schedule-1,,6311e074e749] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130701, count=8, elapsed=1682446570821 2023-05-15 14:05:01.545 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131890, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.547 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131890, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:05:01.542+08:00, endTime=null} 2023-05-15 14:05:01.558 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131891, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.560 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131891, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:05:01.556+08:00, endTime=null} 2023-05-15 14:05:01.568 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,a78ad32873ad] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131890, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116289, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.542+08:00, sub_task_instance_id=131890}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.574 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131888, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.576 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131888, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-05-15T14:05:01.571+08:00, endTime=null} 2023-05-15 14:05:01.577 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:05:01.577 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:05:01.577 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:05:01.577 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:05:01.577 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:05:01.578 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6e4a9bec1ac6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131891, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116290, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.556+08:00, sub_task_instance_id=131891}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.582 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,a78ad32873ad] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131890, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.582 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:05:01.582 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:05:01.582 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:05:01.582 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:05:01.584 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:05:01.593 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131889, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.596 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131889, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-05-15T14:05:01.586+08:00, endTime=null} 2023-05-15 14:05:01.598 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6af42257eb20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131888, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116287, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.571+08:00, sub_task_instance_id=131888}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6af42257eb20] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-05-15 14:05:01.601 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6e4a9bec1ac6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:05:01.602 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6e4a9bec1ac6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:05:01.605 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131894, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.608 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131894, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-05-15T14:05:01.603+08:00, endTime=null} 2023-05-15 14:05:01.609 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:05:01.610 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:05:01.613 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,6af42257eb20] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131888, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.613 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:05:01.614 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,6e4a9bec1ac6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131891, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.616 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,f25dee419941] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131889, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116288, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.586+08:00, sub_task_instance_id=131889}, listMap={cluster_ids=[[2,1,1,1]], tenant_ids=[[6,2,4,3]]}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.616 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,f25dee419941] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [6, 2, 4, 3] 2023-05-15 14:05:01.616 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:05:01.619 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:05:01.622 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131895, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.624 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131895, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-05-15T14:05:01.618+08:00, endTime=null} 2023-05-15 14:05:01.624 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.625 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.625 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.625 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.626 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.626 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.626 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.627 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:05:01.627 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.627 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.627 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,9320af23d7ff] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131894, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116293, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.603+08:00, sub_task_instance_id=131894}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.627 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,9320af23d7ff] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-05-15 14:05:01.627 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.633 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131892, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.633 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.633 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.634 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:05:01.636 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131892, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:05:01.631+08:00, endTime=null} 2023-05-15 14:05:01.641 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:05:01.643 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131895, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116294, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.617+08:00, sub_task_instance_id=131895}, listMap={cluster_ids=[[2,1,1,1]], tenant_ids=[[6,4,2,3]]}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.643 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[6, 4, 2, 3] 2023-05-15 14:05:01.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1684130701643000 2023-05-15 14:05:01.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1684130402257000 2023-05-15 14:05:01.649 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131893, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.650 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131893, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:05:01.647+08:00, endTime=null} 2023-05-15 14:05:01.654 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:05:01.656 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:05:01.657 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b34bc37b631e4cfa,7e97d3787f9a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=b34bc37b631e4cfa, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=ocpcluster,1,false,2,] 2023-05-15 14:05:01.657 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:05:01.658 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b34bc37b631e4cfa,7e97d3787f9a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=b34bc37b631e4cfa, duration=2 ms] 2023-05-15 14:05:01.658 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131892, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116291, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.630+08:00, sub_task_instance_id=131892}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.659 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.660 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.660 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:05:01.663 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131896, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:01.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2023-05-15 14:05:01.681 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131896, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:05:01.661+08:00, endTime=null} 2023-05-15 14:05:01.692 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1684130701686000 2023-05-15 14:05:01.692 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1684130402287000 2023-05-15 14:05:01.692 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.692 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.692 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,90f3be4080094b45,cce941200658] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=90f3be4080094b45, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-05-15 14:05:01.693 INFO 11 --- [ForkJoinPool-66029-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.10, 2982] 2023-05-15 14:05:01.693 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,90f3be4080094b45,cce941200658] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=90f3be4080094b45, duration=1 ms] 2023-05-15 14:05:01.694 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.694 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.694 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.694 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.695 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.695 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.696 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:05:01.697 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,f25dee419941] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131889, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.698 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f2fc7e5ca8a3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131893, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116292, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.647+08:00, sub_task_instance_id=131893}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.698 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f2fc7e5ca8a3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:05:01.698 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.699 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.699 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.700 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.702 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2023-05-15 14:05:01.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:05:01.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:05:01.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:05:01.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:05:01.706 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.706 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.706 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130701000000] 2023-05-15 14:05:01.706 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1684130701703000 2023-05-15 14:05:01.706 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1684130402332000 2023-05-15 14:05:01.707 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.707 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.708 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,bbe19637c95a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131892, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.708 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684128900000000, 1684130701000000] 2023-05-15 14:05:01.710 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,d9e563894a08] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131896, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116295, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:01.661+08:00, sub_task_instance_id=131896}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:01.715 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:05:01.718 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:05:01.719 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1684130701716000 2023-05-15 14:05:01.719 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1684130402308000 2023-05-15 14:05:01.724 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,d9e563894a08] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131896, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.729 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.730 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.731 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:01.732 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.733 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:05:01.735 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:05:01.735 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:05:01.735 INFO 11 --- [pool-3244618-thread-4,fb27d19334114ed7,2789e2f92791] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:05:01.735 INFO 11 --- [pool-3244618-thread-1,fb27d19334114ed7,e9a5c9e3b2a3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:05:01.735 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-2,fb27d19334114ed7,752ac248324b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.736 INFO 11 --- [pool-3244618-thread-3,fb27d19334114ed7,cc9b2e9db135] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:05:01.737 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,86f16c4fad61] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131895, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.779 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.780 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.780 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.780 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.780 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.781 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.11, 2982] 2023-05-15 14:05:01.781 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:01.783 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.784 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.786 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.13, 2982] 2023-05-15 14:05:01.786 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.786 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.788 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.788 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.788 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:01.788 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:01.788 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:05:01.789 INFO 11 --- [ForkJoinPool-66029-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.12, 2982] 2023-05-15 14:05:01.792 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:05:01.798 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:05:01.798 INFO 11 --- [pool-3244618-thread-5,fb27d19334114ed7,2cbb93fafe5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:05:01.803 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:05:01.803 INFO 11 --- [pool-3244618-thread-6,fb27d19334114ed7,c0dde77b8ef3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:05:01.845 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f2fc7e5ca8a3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:05:01.857 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f2fc7e5ca8a3] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131893, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:01.877 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c3c4b4e039b048fb,311b20e71207] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=c3c4b4e039b048fb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:01.883 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c3c4b4e039b048fb,311b20e71207] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=c3c4b4e039b048fb, duration=6 ms] 2023-05-15 14:05:01.940 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116287, status:SUCCESS 2023-05-15 14:05:01.942 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116287, status:SUCCESS 2023-05-15 14:05:01.953 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116288, status:SUCCESS 2023-05-15 14:05:01.954 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116288, status:SUCCESS 2023-05-15 14:05:01.964 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116289, status:SUCCESS 2023-05-15 14:05:01.965 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116289, status:SUCCESS 2023-05-15 14:05:01.992 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116290, status:SUCCESS 2023-05-15 14:05:01.994 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116290, status:SUCCESS 2023-05-15 14:05:02.005 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116291, status:SUCCESS 2023-05-15 14:05:02.007 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116291, status:SUCCESS 2023-05-15 14:05:02.018 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116292, status:SUCCESS 2023-05-15 14:05:02.019 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116292, status:SUCCESS 2023-05-15 14:05:02.024 INFO 11 --- [pool-3244617-thread-1,fb27d19334114ed7,676a9f2a51c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:05:02.034 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116294, status:SUCCESS 2023-05-15 14:05:02.035 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116294, status:SUCCESS 2023-05-15 14:05:02.042 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116295, status:SUCCESS 2023-05-15 14:05:02.043 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116295, status:SUCCESS 2023-05-15 14:05:02.079 WARN 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obcluster, ob tenant id:1001 2023-05-15 14:05:02.086 INFO 11 --- [pool-3244617-thread-2,fb27d19334114ed7,02287c06b90c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:05:02.087 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,9320af23d7ff] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-05-15 14:05:02.091 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,9320af23d7ff] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131894, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:02.755 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:02.756 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:03.078 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116293, status:SUCCESS 2023-05-15 14:05:03.079 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116293, status:SUCCESS 2023-05-15 14:05:04.714 INFO 11 --- [pool-ocp-schedules-8,,48277fb807d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:05:04.722 INFO 11 --- [pool-ocp-schedules-8,,48277fb807d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:05:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:04.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:04.827808+08:00, lastProcessTime=2023-05-15T14:04:54.829981+08:00 2023-05-15 14:05:04.864 INFO 11 --- [alarm-detect-5,,28af99d38ad6] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130684 ~ 1684130694 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130690=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:04.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:04.827808+08:00 2023-05-15 14:05:05.757 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:05.757 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:05.976 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:05.982 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:05.987 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:05.992 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:06.025 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:06.0, nextRun=2023-05-15 14:05:05.0, currentDate=Mon May 15 14:05:06 CST 2023 2023-05-15 14:05:06.025 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:04:06.0, nextRunTime=2023-05-15 14:05:05.0} 2023-05-15 14:05:06.029 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:05:06.029 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:06.029 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:06.770 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131897, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:06.772 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131897, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:05:06.768+08:00, endTime=null} 2023-05-15 14:05:06.791 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,5b74e47c3f59] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131897, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116296, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:06.768+08:00, sub_task_instance_id=131897}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:06.819 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,1535caae9708] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:05:06.819 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,174bc5d946d7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:05:06.819 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,963f7c8215ee] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:05:06.854 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,1535caae9708] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:05:06.855 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,963f7c8215ee] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:05:06.860 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,174bc5d946d7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:05:06.872 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,5b74e47c3f59] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131897, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:06.966 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:07.170 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116296, status:SUCCESS 2023-05-15 14:05:07.171 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116296, status:SUCCESS 2023-05-15 14:05:08.115 INFO 11 --- [pool-ocp-schedules-4,,6a958b49b0e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:08.264 INFO 11 --- [pool-overview-container-8,,02e93a61dfc0] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:05:08.264 INFO 11 --- [pool-overview-container-8,,02e93a61dfc0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:08.511 INFO 11 --- [pool-scanner--15972267-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:05:08.758 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:08.759 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:08.871 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,b2154b9240954577,a52e32fffdfc] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames?groupBy=database_name, client=192.168.80.229, traceId=b2154b9240954577, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrendsSubLevelNames(Long,Long,String,String), args=2,6,database_name,] 2023-05-15 14:05:08.871 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,b2154b9240954577,a52e32fffdfc] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=null, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:05:08.874 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d9a9abd74d4663,ec3b402e2052] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames?groupBy=tablegroup_name, client=192.168.80.229, traceId=00d9a9abd74d4663, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrendsSubLevelNames(Long,Long,String,String), args=2,6,tablegroup_name,] 2023-05-15 14:05:08.874 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d9a9abd74d4663,ec3b402e2052] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[], labels={}, interval=null, minStep=null, groupBy=[tablegroup_name], limit=null, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:05:08.875 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,f493098745264074,316cf34ee5f3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=f493098745264074, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:05:08.875 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,f493098745264074,316cf34ee5f3] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:05:08.888 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1a5701c14f6847c7,7a3968b7e293] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A05%3A08%2B08%3A00&endTime=2023-05-15T14%3A05%3A08%2B08%3A00&metrics=ob_tenant_min_memory%2Cob_tenant_max_memory_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=1a5701c14f6847c7, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:05:08+08:00,2023-05-15T14:05:08+08:00,ob_tenant_min_memory,ob_tenant_max_memory_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:08.888 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8d5b5cbe37934a5e,e0011dbfb76a] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A05%3A08%2B08%3A00&endTime=2023-05-15T14%3A05%3A08%2B08%3A00&metrics=ob_tenant_min_cpu%2Cob_tenant_max_cpu_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=8d5b5cbe37934a5e, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:05:08+08:00,2023-05-15T14:05:08+08:00,ob_tenant_min_cpu,ob_tenant_max_cpu_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:08.889 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1a5701c14f6847c7,7a3968b7e293] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:05:08+08:00, endTime=2023-05-15T14:05:08+08:00, metrics=[ob_tenant_min_memory, ob_tenant_max_memory_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:05:08.889 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5a1e3dd94b54ca3,c068686b6c35] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends?startTime=2023-05-08T14%3A05%3A08%2B08%3A00&endTime=2023-05-15T14%3A05%3A08%2B08%3A00&metrics=ob_tenant_data_size%2Cob_tenant_disk_used%2Cob_tenant_disk_used_percentage&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=d5a1e3dd94b54ca3, method=IterableResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTrends(Long,Long,OffsetDateTime,OffsetDateTime,String,String,String), args=2,6,2023-05-08T14:05:08+08:00,2023-05-15T14:05:08+08:00,ob_tenant_data_size,ob_tenant_disk_used,ob_tenant_disk_used_percentage,app,obregion,ob_cluster_id,tenant_name,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:08.889 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8d5b5cbe37934a5e,e0011dbfb76a] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:05:08+08:00, endTime=2023-05-15T14:05:08+08:00, metrics=[ob_tenant_min_cpu, ob_tenant_max_cpu_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:05:08.889 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5a1e3dd94b54ca3,c068686b6c35] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTrends, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=2023-05-08T14:05:08+08:00, endTime=2023-05-15T14:05:08+08:00, metrics=[ob_tenant_data_size, ob_tenant_disk_used, ob_tenant_disk_used_percentage], labels={app=OB, tenant_name=test_tenant, obregion=obcluster, ob_cluster_id=1}, interval=null, minStep=null, groupBy=[app, obregion, ob_cluster_id, tenant_name], limit=null, prometheusLabels=[OcpPrometheusLabel(labelKey=app, labelValue=OB, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=obregion, labelValue=obcluster, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=ob_cluster_id, labelValue=1, labelMatch=MATCH_EQ), OcpPrometheusLabel(labelKey=tenant_name, labelValue=test_tenant, labelMatch=MATCH_EQ)], maxPoints=1440) 2023-05-15 14:05:08.897 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,b2154b9240954577,a52e32fffdfc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames client=192.168.80.229, traceId=b2154b9240954577, duration=28 ms] 2023-05-15 14:05:08.904 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d9a9abd74d4663,ec3b402e2052] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends/subLevelNames client=192.168.80.229, traceId=00d9a9abd74d4663, duration=32 ms] 2023-05-15 14:05:08.905 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,f493098745264074,316cf34ee5f3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=f493098745264074, duration=33 ms] 2023-05-15 14:05:08.911 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1a5701c14f6847c7,7a3968b7e293] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:05:08.914 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8d5b5cbe37934a5e,e0011dbfb76a] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:05:08.935 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5a1e3dd94b54ca3,c068686b6c35] o.o.o.i.t.TenantResourceTrendServiceImpl : tenant scope query, query from metric, tenantId=6 2023-05-15 14:05:08.953 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,0a0adb7ff7af4b24,9c538260315d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=0a0adb7ff7af4b24, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:05:08.953 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,0a0adb7ff7af4b24,9c538260315d] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:05:08.968 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,0a0adb7ff7af4b24,9c538260315d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=0a0adb7ff7af4b24, duration=16 ms] 2023-05-15 14:05:08.991 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5ea994af49754d19,6fc4906bdce7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop?limit=10&metrics=ob_tenant_data_size&groupBy=database_name&labels=, client=192.168.80.229, traceId=5ea994af49754d19, method=SuccessResponse com.oceanbase.ocp.server.controller.ObResourceController.tenantResourceTop(Long,Long,String,String,String,Integer), args=2,6,ob_tenant_data_size,database_name,,10,] 2023-05-15 14:05:08.991 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5ea994af49754d19,6fc4906bdce7] o.o.o.i.t.TenantResourceTrendServiceImpl : resourceTop, clusterId=2, tenantId=6, param=MetricQueryParams(startTime=null, endTime=null, metrics=[ob_tenant_data_size], labels={}, interval=null, minStep=null, groupBy=[database_name], limit=10, prometheusLabels=[], maxPoints=1440) 2023-05-15 14:05:08.998 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5a1e3dd94b54ca3,c068686b6c35] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=d5a1e3dd94b54ca3, duration=111 ms] 2023-05-15 14:05:09.010 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5ea994af49754d19,6fc4906bdce7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTop client=192.168.80.229, traceId=5ea994af49754d19, duration=21 ms] 2023-05-15 14:05:09.030 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1a5701c14f6847c7,7a3968b7e293] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=1a5701c14f6847c7, duration=144 ms] 2023-05-15 14:05:09.054 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8d5b5cbe37934a5e,e0011dbfb76a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/resourceTrends client=192.168.80.229, traceId=8d5b5cbe37934a5e, duration=168 ms] 2023-05-15 14:05:09.946 INFO 11 --- [pool-ocp-schedules-15,,5b162a53f573] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5390,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40050} 2023-05-15 14:05:09.947 INFO 11 --- [pool-ocp-schedules-15,,5b162a53f573] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:05:10.264 INFO 11 --- [pool-ocp-schedules-5,,5ee881b6f2cb] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:05:10.633 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,2bc6890149c3468a,98a94f909b12] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&clusterId=2, client=192.168.80.229, traceId=2bc6890149c3468a, method=PaginatedResponse com.oceanbase.ocp.server.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 2000, sort: id: ASC],NORMAL,TENANT,2,] 2023-05-15 14:05:10.761 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,2bc6890149c3468a,98a94f909b12] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=192.168.80.229, traceId=2bc6890149c3468a, duration=130 ms] 2023-05-15 14:05:10.909 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,1df7e6c527364593,b4bbd3f10823] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=sql_all_rt%2Csql_select_rt%2Csql_insert_rt%2Csql_update_rt%2Csql_replace_rt%2Csql_delete_rt%2Csql_other_rt&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=1df7e6c527364593, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,sql_all_rt,sql_select_rt,sql_insert_rt,sql_update_rt,sql_replace_rt,sql_delete_rt,sql_other_rt,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:10.931 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,069d1a3dbcef4157,2f6c082df45f] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=sql_all_count%2Csql_select_count%2Csql_insert_count%2Csql_update_count%2Csql_replace_count%2Csql_delete_count%2Csql_other_count&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=069d1a3dbcef4157, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,sql_all_count,sql_select_count,sql_insert_count,sql_update_count,sql_replace_count,sql_delete_count,sql_other_count,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:10.937 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,5f5382f4d52c417d,0394cf44635a] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=sql_local_count%2Csql_remote_count%2Csql_distributed_count&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=5f5382f4d52c417d, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,sql_local_count,sql_remote_count,sql_distributed_count,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:10.942 INFO 11 --- [pool-scanner--15972279-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.961 INFO 11 --- [pool-scanner--15972293-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.963 INFO 11 --- [pool-scanner--15972287-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.964 INFO 11 --- [pool-scanner--15972286-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.964 INFO 11 --- [pool-scanner--15972285-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.964 INFO 11 --- [pool-scanner--15972281-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.964 INFO 11 --- [pool-scanner--15972289-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.965 INFO 11 --- [pool-scanner--15972300-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.965 INFO 11 --- [pool-scanner--15972294-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.965 INFO 11 --- [pool-scanner--15972298-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.972 INFO 11 --- [pool-scanner--15972301-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.972 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,3f8e48b961bc43e1,cef6c64d5b86] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=sql_local_time%2Csql_remote_time%2Csql_distributed_time&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=3f8e48b961bc43e1, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,sql_local_time,sql_remote_time,sql_distributed_time,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:10.972 INFO 11 --- [pool-scanner--15972288-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.972 INFO 11 --- [pool-scanner--15972283-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.973 INFO 11 --- [pool-scanner--15972284-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.974 INFO 11 --- [pool-scanner--15972295-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.974 INFO 11 --- [pool-scanner--15972280-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.974 INFO 11 --- [pool-scanner--15972282-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.977 INFO 11 --- [pool-scanner--15972292-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.978 INFO 11 --- [pool-scanner--15972297-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.980 INFO 11 --- [pool-scanner--15972302-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.980 INFO 11 --- [pool-scanner--15972290-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.982 INFO 11 --- [pool-scanner--15972304-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.982 INFO 11 --- [pool-scanner--15972291-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.983 INFO 11 --- [pool-scanner--15972296-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.986 INFO 11 --- [pool-scanner--15972316-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.987 INFO 11 --- [pool-scanner--15972313-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.987 INFO 11 --- [pool-scanner--15972317-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.988 INFO 11 --- [pool-scanner--15972299-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.988 INFO 11 --- [pool-scanner--15972306-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.988 INFO 11 --- [pool-scanner--15972307-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.988 INFO 11 --- [pool-scanner--15972314-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.988 INFO 11 --- [pool-scanner--15972308-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.989 INFO 11 --- [pool-scanner--15972312-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.989 INFO 11 --- [pool-scanner--15972310-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.989 INFO 11 --- [pool-scanner--15972305-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.990 INFO 11 --- [pool-scanner--15972309-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.991 INFO 11 --- [pool-scanner--15972303-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.991 INFO 11 --- [pool-scanner--15972311-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.991 INFO 11 --- [pool-scanner--15972315-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.992 INFO 11 --- [pool-scanner--15972318-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.993 INFO 11 --- [pool-scanner--15972319-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:10.994 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,5f5382f4d52c417d,0394cf44635a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=5f5382f4d52c417d, duration=59 ms] 2023-05-15 14:05:11.017 INFO 11 --- [pool-scanner--15972320-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.039 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,3f8e48b961bc43e1,cef6c64d5b86] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=3f8e48b961bc43e1, duration=76 ms] 2023-05-15 14:05:11.057 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,069d1a3dbcef4157,2f6c082df45f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=069d1a3dbcef4157, duration=137 ms] 2023-05-15 14:05:11.076 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,1df7e6c527364593,b4bbd3f10823] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=1df7e6c527364593, duration=170 ms] 2023-05-15 14:05:11.440 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f81f8dfc2fee4ec7,d63be282c22d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=wait_event_count%2Csystem_event_internal_total_waits%2Csystem_event_sync_rpc_total_waits%2Csystem_event_row_lock_wait_total_waits%2Csystem_event_io_total_waits%2Csystem_event_latch_total_waits%2Csystem_event_other_total_waits&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=f81f8dfc2fee4ec7, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,wait_event_count,system_event_internal_total_waits,system_event_sync_rpc_total_waits,system_event_row_lock_wait_total_waits,system_event_io_total_waits,system_event_latch_total_waits,system_event_other_total_waits,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:11.457 INFO 11 --- [pool-scanner--15972325-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.457 INFO 11 --- [pool-scanner--15972323-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.459 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,5d360a72e8874d3d,4c4904ec38bb] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-15T13%3A05%3A10%2B08%3A00&endTime=2023-05-15T14%3A05%3A10%2B08%3A00&metrics=active_session%2Call_session&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Atest_tenant&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=5d360a72e8874d3d, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-15T13:05:10+08:00,2023-05-15T14:05:10+08:00,active_session,all_session,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:test_tenant,] 2023-05-15 14:05:11.464 INFO 11 --- [pool-scanner--15972324-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.464 INFO 11 --- [pool-scanner--15972326-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.466 INFO 11 --- [pool-scanner--15972322-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.467 INFO 11 --- [pool-scanner--15972321-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.467 INFO 11 --- [pool-scanner--15972327-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.474 INFO 11 --- [pool-scanner--15972329-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.475 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f81f8dfc2fee4ec7,d63be282c22d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=f81f8dfc2fee4ec7, duration=37 ms] 2023-05-15 14:05:11.482 INFO 11 --- [pool-scanner--15972328-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=168, nodesFilledCount=56 2023-05-15 14:05:11.487 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,5d360a72e8874d3d,4c4904ec38bb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=5d360a72e8874d3d, duration=33 ms] 2023-05-15 14:05:11.680 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,4aff97a609c949a1,365f38008b62] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=4aff97a609c949a1, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:11.687 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,4aff97a609c949a1,365f38008b62] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=4aff97a609c949a1, duration=8 ms] 2023-05-15 14:05:11.759 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:11.760 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:11.829 INFO 11 --- [pool-ocp-schedules-14,,721fac74d2e0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:11.829 INFO 11 --- [pool-ocp-schedules-14,,721fac74d2e0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:11.906 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,2c206261ec0f4dcf,7741d2bc175d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=2c206261ec0f4dcf, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:11.911 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,2c206261ec0f4dcf,7741d2bc175d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=2c206261ec0f4dcf, duration=5 ms] 2023-05-15 14:05:12.373 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:13.068 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,653dfd61f15d4e76,12720ef3fef1] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch?id=2&tenantId=6, client=192.168.80.229, traceId=653dfd61f15d4e76, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitch(Long,Long), args=2,6,] 2023-05-15 14:05:13.070 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/performance/config?id=2&tenantId=6, client=192.168.80.229, traceId=8eaa10ae6b0c46bf, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getClusterPerfConfig(Long,Long), args=2,6,] 2023-05-15 14:05:13.070 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,77634d66ff1c4d51,294e3c526f75] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events?null, client=192.168.80.229, traceId=77634d66ff1c4d51, method=PaginatedResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchEvents(Long,Long,OffsetDateTime,OffsetDateTime,Pageable), args=2,6,Page request [number: 0, size 2000, sort: timestamp: ASC],] 2023-05-15 14:05:13.091 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,77634d66ff1c4d51,294e3c526f75] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events client=192.168.80.229, traceId=77634d66ff1c4d51, duration=23 ms] 2023-05-15 14:05:13.102 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.103 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.103 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.104 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:13.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,653dfd61f15d4e76,12720ef3fef1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch client=192.168.80.229, traceId=653dfd61f15d4e76, duration=42 ms] 2023-05-15 14:05:13.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() REGEXP 'OceanBase.*?CE' 2023-05-15 14:05:13.111 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.111 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.111 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2023-05-15 14:05:13.122 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/outlines?null, client=192.168.80.229, traceId=c927c22d35f84d94, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObOutlineController.getSqlOutline(Long,Long,String,String,OffsetDateTime,OffsetDateTime,Boolean), args=2,6,] 2023-05-15 14:05:13.150 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:05:13.150 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:05:13.165 ERROR 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573862) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573862) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:13.167 ERROR 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] com.alibaba.druid.pool.DruidDataSource : {dataSource-94} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573862) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573862) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:13.167 INFO 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] com.alibaba.druid.pool.DruidDataSource : {dataSource-94,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:05:13.168 INFO 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] com.alibaba.druid.pool.DruidDataSource : {dataSource-94} closing ... 2023-05-15 14:05:13.171 INFO 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] com.alibaba.druid.pool.DruidDataSource : {dataSource-94} closed 2023-05-15 14:05:13.171 ERROR 11 --- [obsdk-init-thread-11,c927c22d35f84d94,2fab9c2f8503] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573862) Get Location Cache Fail 2023-05-15 14:05:13.172 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573862) Get Location Cache Fail 2023-05-15 14:05:13.172 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573862) Get Location Cache Fail 2023-05-15 14:05:13.173 WARN 11 --- [http-nio-0.0.0.0-8080-exec-64,c927c22d35f84d94,7931597f54ba] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/outlines client=192.168.80.229, traceId=c927c22d35f84d94, duration=52 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:05:13.186 ERROR 11 --- [Druid-ConnectionPool-Create-1511578641,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573865) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573865) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:05:13.221 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A35%3A12%2B08%3A00&endTime=2023-05-15T14%3A05%3A12%2B08%3A00&inner=false, client=192.168.80.229, traceId=2040bcd1ed06415b, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:35:12+08:00,2023-05-15T14:05:12+08:00,false,] 2023-05-15 14:05:13.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,a92dd6a62e0045ba,a3b84b37c1c2] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents?startTime=2023-05-15T13%3A35%3A12%2B08%3A00&endTime=2023-05-15T14%3A05%3A12%2B08%3A00, client=192.168.80.229, traceId=a92dd6a62e0045ba, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchOffIntervalLimit3(Long,Long,OffsetDateTime,OffsetDateTime), args=2,6,2023-05-15T13:35:12+08:00,2023-05-15T14:05:12+08:00,] 2023-05-15 14:05:13.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:05:13.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:05:13.240 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,a92dd6a62e0045ba,a3b84b37c1c2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents client=192.168.80.229, traceId=a92dd6a62e0045ba, duration=20 ms] 2023-05-15 14:05:13.246 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.246 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.246 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.247 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:13.252 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.252 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.253 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:13.257 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:35:12, end=2023-05-15T14:05:12}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:05:13.260 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:35:12, end=2023-05-15T14:05:12}} 2023-05-15 14:05:13.263 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.263 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.264 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2023-05-15 14:05:13.314 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:05:13.320 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:05:13.320 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:05:13.329 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,2040bcd1ed06415b,9432e04561e2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=2040bcd1ed06415b, duration=109 ms] 2023-05-15 14:05:13.356 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:13.356 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:13.357 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:05:13.582 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,8eaa10ae6b0c46bf,cee2cd17e702] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/performance/config client=192.168.80.229, traceId=8eaa10ae6b0c46bf, duration=514 ms] 2023-05-15 14:05:14.408 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130640, nodeCount=178, affectRows=178 2023-05-15 14:05:14.409 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130640, nodeCount=522, affectRows=522 2023-05-15 14:05:14.411 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130640, nodeCount=1000, affectRows=1000 2023-05-15 14:05:14.411 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130640, nodeCount=178, validCount=178 2023-05-15 14:05:14.415 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130640, nodeCount=522, validCount=522 2023-05-15 14:05:14.418 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130640, nodeCount=1000, affectRows=1000 2023-05-15 14:05:14.419 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130640, nodeCount=1000, validCount=1000 2023-05-15 14:05:14.428 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130640, nodeCount=1000, validCount=1000 2023-05-15 14:05:14.760 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:14.761 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:14.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:14.831975+08:00, lastProcessTime=2023-05-15T14:05:04.827808+08:00 2023-05-15 14:05:14.864 INFO 11 --- [alarm-detect-3,,b8e3c56205e6] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130694 ~ 1684130704 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130700=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:14.903 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:14.831975+08:00 2023-05-15 14:05:14.911 INFO 11 --- [pool-ocp-schedules-7,,2bde76e84d68] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:05:14.912 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:05:14.912 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:05:14.912 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:05:14.912 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:05:14.912 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:05:14.917 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:05:14.918 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:05:14.918 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:05:14.918 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:05:14.919 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:05:14.946 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:05:14.946 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:05:14.949 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:05:14.949 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:05:14.948741107+08:00 2023-05-15 14:05:14.949 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:05:14.949+08:00 2023-05-15 14:05:14.949 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:05:14.948819548+08:00 2023-05-15 14:05:14.949 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:05:14.949+08:00 2023-05-15 14:05:14.951 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:05:14.950751613+08:00 2023-05-15 14:05:14.951 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:05:14.951 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:05:14.951+08:00 2023-05-15 14:05:14.951 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:05:14.953 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:05:14.949+08:00 and status AVAILABLE 2023-05-15 14:05:14.953 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:05:14.953095121+08:00 2023-05-15 14:05:14.953 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:05:14.953+08:00 2023-05-15 14:05:14.953 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:05:14.953193213+08:00 2023-05-15 14:05:14.954 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:05:14.953+08:00 2023-05-15 14:05:14.954 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:05:14.949+08:00 and status AVAILABLE 2023-05-15 14:05:14.954 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:05:14.951+08:00 and status AVAILABLE 2023-05-15 14:05:14.956 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:05:14.953+08:00 and status AVAILABLE 2023-05-15 14:05:14.958 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:05:14.953+08:00 and status AVAILABLE 2023-05-15 14:05:14.993 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:05:14.993 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:05:14.994 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:05:14.994 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:05:14.994 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:05:15.025 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130714 0 0 , error= 2023-05-15 14:05:15.025 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130714 0 0 , error= 2023-05-15 14:05:15.028 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130714 0 0 , error= 2023-05-15 14:05:15.028 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130714 0 0 , error= 2023-05-15 14:05:15.031 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:05:15.031+08:00 2023-05-15 14:05:15.031 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:05:15.031+08:00 2023-05-15 14:05:15.032 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:05:15.031+08:00 2023-05-15 14:05:15.032 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:05:15.031+08:00 2023-05-15 14:05:15.036 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:05:15.031+08:00 and status AVAILABLE 2023-05-15 14:05:15.036 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:05:15.031+08:00 and status AVAILABLE 2023-05-15 14:05:15.036 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/63577C163A26A8D3FC2EA6A2C92C4E83/text?startTime=2023-05-15T13%3A35%3A12%2B08%3A00&endTime=2023-05-15T14%3A05%3A12%2B08%3A00, client=192.168.80.229, traceId=9f7f26c136eb4e6c, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,63577C163A26A8D3FC2EA6A2C92C4E83,2023-05-15T13:35:12+08:00,2023-05-15T14:05:12+08:00,] 2023-05-15 14:05:15.037 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:05:15.031+08:00 and status AVAILABLE 2023-05-15 14:05:15.037 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:05:15.031+08:00 and status AVAILABLE 2023-05-15 14:05:15.039 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:05:15.039 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:05:15.040 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:05:15.040 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:05:15.049 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:05:15.050 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:05:15.051 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:05:15.051 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:05:15.063 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.064 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.064 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:15.065 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:15.069 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.069 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:15.070 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:15.074 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:05:15.077 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:05:15.078 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:15.079 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.080 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:05:15.080 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.080 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:15.080 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:05:15.080 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:15.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:15.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:15.086 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:05:15.127 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:05:15.134 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:05:15.135 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:05:15.144 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:05:15.148 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:05:15.149 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:05:15.157 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:05:15.158 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,9f7f26c136eb4e6c,8c8ffb589e88] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/63577C163A26A8D3FC2EA6A2C92C4E83/text client=192.168.80.229, traceId=9f7f26c136eb4e6c, duration=124 ms] 2023-05-15 14:05:15.167 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:05:15.185 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:05:15.186 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:05:15.186 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:05:15.187 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:05:15.236 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:05:15.240 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:05:15.242 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:05:15.250 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:05:15.254 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:05:15.254 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:05:15.265 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:05:15.278 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:05:15.294 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:05:15.295 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:05:15.297 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:05:15.309 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:05:15.339 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:05:15.352 INFO 11 --- [pool-Check host status12,,fac8029508a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:05:15.354 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:05:15.354 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:05:15.368 INFO 11 --- [pool-Check host status10,,9b49d363ff66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:05:15.369 INFO 11 --- [pool-Check host status9,,61f9b3fe25ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:05:15.401 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:05:15.413 INFO 11 --- [pool-Check host status26,,04aca716b00d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:05:15.954 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130715 0 0 , error= 2023-05-15 14:05:15.954 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:05:15.954+08:00 2023-05-15 14:05:15.959 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:05:15.954+08:00 and status AVAILABLE 2023-05-15 14:05:15.971 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:05:15.982 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:15.985 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:05:15.987 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:15.992 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:16.002 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:16.022 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:05:16.082 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:05:16.096 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:05:16.121 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:05:16.179 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:05:16.193 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:05:16.213 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:05:16.280 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:05:16.297 INFO 11 --- [pool-Check host status22,,ae6f449d5fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:05:16.318 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:05:16.318 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:05:16.318 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:05:16.318 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:05:16.318 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:05:16.325 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:05:16.325 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:05:16.325 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:05:16.325 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:05:16.326 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:05:16.351 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:05:16.352 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:05:16.355 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:05:16.356 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:05:16.356 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:05:16.371 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:05:16.372 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:05:16.372 INFO 11 --- [pool-Check host status16,,73d74306a3eb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:05:16.372 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:05:16.372 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:05:16.372 INFO 11 --- [pool-Check host status25,,23ae07071531] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:05:16.373 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:05:16.373 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:05:16.373 INFO 11 --- [pool-Check host status15,,43e4ecf71405] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:05:16.377 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:05:16.377 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:05:16.377 INFO 11 --- [pool-Check host status24,,b6352945e6ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:05:16.378 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:05:16.378 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:05:16.378 INFO 11 --- [pool-Check host status18,,40c76c309550] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:05:16.969 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:05:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:17.630 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:17.630 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:17.630 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:17.633 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:17.633 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:17.633 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:17.662 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:17.663 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:17.668 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:17.668 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:17.669 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:17.672 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:05:17.672 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:05:17.672 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:05:17.672 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:05:17.676 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:02:48, target=2023-05-15T14:00:47, granule=120000ms, delay=121000ms) 2023-05-15 14:05:17.676 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:03:02, target=2023-05-15T14:00:47, granule=120000ms, delay=135000ms) 2023-05-15 14:05:17.676 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:51:17, granule=600000ms, delay=570000ms) 2023-05-15 14:05:17.676 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:53:17, granule=600000ms, delay=450000ms) 2023-05-15 14:05:17.677 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:02:48, target=2023-05-15T14:00:47, granule=120000ms, delay=121000ms) 2023-05-15 14:05:17.677 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:00:47, target=2023-05-15T13:51:17, granule=600000ms, delay=570000ms) 2023-05-15 14:05:17.677 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=90ms, period=120000ms 2023-05-15 14:05:17.762 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:17.763 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:18.135 INFO 11 --- [pool-ocp-schedules-15,,4580d930dc7e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:18.263 INFO 11 --- [pool-overview-container-5,,ec2cee9f9f03] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:18.514 INFO 11 --- [pool-scanner--15972429-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:20.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:20.0, nextRun=2023-05-15 14:05:20.0, currentDate=Mon May 15 14:05:20 CST 2023 2023-05-15 14:05:20.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:04:20.0, nextRunTime=2023-05-15 14:05:20.0} 2023-05-15 14:05:20.616 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:05:20.616 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:20.616 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:20.763 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:20.764 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:20.990 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131898, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:20.992 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131898, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:05:20.987+08:00, endTime=null} 2023-05-15 14:05:21.017 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,47bc623cc6b8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131898, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116297, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:20.987+08:00, sub_task_instance_id=131898}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:21.019 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,47bc623cc6b8] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:05:21.020 INFO 11 --- [ForkJoinPool-66032-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:05:21.020 INFO 11 --- [ForkJoinPool-66032-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:05:21.027 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,47bc623cc6b8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131898, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:21.469 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116297, status:SUCCESS 2023-05-15 14:05:21.469 INFO 11 --- [pool-scanner--15972430-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:21.470 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116297, status:SUCCESS 2023-05-15 14:05:21.490 INFO 11 --- [pool-scanner--15972431-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:21.508 INFO 11 --- [pool-scanner--15972432-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:21.533 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.542 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.582 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.586 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.600 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.605 INFO 11 --- [pool-ocp-schedules-15,,1433d433d67a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:21.740 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-05-15 14:05:21.769 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:05:21.795 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:05:21.812 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:21.815 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:21.816 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:21.818 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:05:21.844 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:05:21.864 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:05:21.889 INFO 11 --- [pool-ocp-schedules-16,,b578946d8eab] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=5 2023-05-15 14:05:23.764 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:23.765 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:24.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:24.829232+08:00, lastProcessTime=2023-05-15T14:05:14.831975+08:00 2023-05-15 14:05:24.919 INFO 11 --- [alarm-detect-3,,62ab3c80a0f8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130704 ~ 1684130714 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130710=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:24.938 INFO 11 --- [alarm-detect-1,,6a5621770024] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130654 ~ 1684130714 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389097.988, values={1684130700=389097.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:05:24.939 INFO 11 --- [alarm-detect-1,,6a5621770024] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130654 ~ 1684130714 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993895.358, values={1684130700=993895.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:24.965 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:24.829232+08:00 2023-05-15 14:05:25.766 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:26.0, nextRun=2023-05-15 14:05:25.0, currentDate=Mon May 15 14:05:25 CST 2023 2023-05-15 14:05:25.766 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:04:26.0, nextRunTime=2023-05-15 14:05:25.0} 2023-05-15 14:05:25.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:05:25.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:25.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:25.988 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:25.990 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:25.995 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:26.004 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:26.079 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131899, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:26.081 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131899, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:05:26.076+08:00, endTime=null} 2023-05-15 14:05:26.100 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131899, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116298, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:26.076+08:00, sub_task_instance_id=131899}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:05:26.139 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:05:26.140 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:05:26.140 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:05:26.141 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:05:26.141 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:05:26.146 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6c5d25fda22c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131899, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:26.569 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116298, status:SUCCESS 2023-05-15 14:05:26.570 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116298, status:SUCCESS 2023-05-15 14:05:26.765 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:26.767 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:26.972 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:27.375 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:27.516 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e98cb2f740444fc4,4097ebc393e8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=e98cb2f740444fc4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:05:27.518 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e98cb2f740444fc4,4097ebc393e8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e98cb2f740444fc4, duration=2 ms] 2023-05-15 14:05:28.156 INFO 11 --- [pool-ocp-schedules-9,,5e6c1c9e717f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:28.263 INFO 11 --- [pool-overview-container-3,,e8b684b084b3] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:28.512 INFO 11 --- [pool-scanner--15972530-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:29.713 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,6d0154760d2f4340,11bac9989215] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=6d0154760d2f4340, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:29.719 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,6d0154760d2f4340,11bac9989215] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=6d0154760d2f4340, duration=6 ms] 2023-05-15 14:05:29.767 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:29.768 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:30.284 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.317 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.320 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.322 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.324 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.326 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.327 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.329 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.331 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.344 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.352 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.357 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.363 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.367 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.374 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.378 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.382 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.384 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.385 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.386 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.387 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.388 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.390 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.391 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.392 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T13:59:30.156, end=2023-05-15T14:04:30.281} 2023-05-15 14:05:30.392 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:05:30.943 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:31.0, nextRun=2023-05-15 14:05:30.0, currentDate=Mon May 15 14:05:30 CST 2023 2023-05-15 14:05:30.943 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:04:31.0, nextRunTime=2023-05-15 14:05:30.0} 2023-05-15 14:05:30.951 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:05:30.951 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:30.952 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:30.952 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:30.965 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:05:30.0, currentDate=Mon May 15 14:05:30 CST 2023 2023-05-15 14:05:30.965 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:05:30.0} 2023-05-15 14:05:30.970 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:05:30.970 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:05:30.970 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:05:31.147 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131902, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:31.149 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131902, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:05:31.144+08:00, endTime=null} 2023-05-15 14:05:31.162 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131900, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:31.167 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131900, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:05:31.159+08:00, endTime=null} 2023-05-15 14:05:31.171 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d3ac612f2329] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131902, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116300, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:31.143+08:00, sub_task_instance_id=131902}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:05:31.171 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d3ac612f2329] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:05:31.176 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131901, operation:EXECUTE, state: RUNNING 2023-05-15 14:05:31.178 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131901, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:05:31.174+08:00, endTime=null} 2023-05-15 14:05:31.192 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131900, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116299, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:31.159+08:00, sub_task_instance_id=131900}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:05:31.193 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:05:31.194 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:05:31.194 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:05:31.194 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:05:31.194 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:05:31.194 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:05:31.195 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131901, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116299, task_operation=execute, latest_execution_start_time=2023-05-15T14:05:31.174+08:00, sub_task_instance_id=131901}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:05:31.206 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.206 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.206 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.207 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.207 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.207 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.208 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.209 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:05:31.210 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.211 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.211 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.211 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:05:31.212 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.212 INFO 11 --- [pool-3244785-thread-2,fb27d19334114ed7,22344d0591d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:05:31.212 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:05:31.212 INFO 11 --- [pool-3244785-thread-1,fb27d19334114ed7,7e32ebba4d3d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:05:31.213 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:05:31.213 INFO 11 --- [pool-3244785-thread-3,fb27d19334114ed7,9d269b408f33] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:05:31.213 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:05:31.213 INFO 11 --- [pool-3244785-thread-4,fb27d19334114ed7,17a317c05951] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:05:31.213 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.214 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.214 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.215 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.217 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.217 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.218 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:05:31.219 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.219 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.220 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:05:31.223 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:05:31.223 INFO 11 --- [pool-3244785-thread-6,fb27d19334114ed7,cbc1645c9cd1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:05:31.225 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:05:31.225 INFO 11 --- [pool-3244785-thread-5,fb27d19334114ed7,90123dec0e17] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:05:31.226 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d3ac612f2329] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:05:31.230 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d3ac612f2329] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131902, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:31.232 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.232 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.232 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.232 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.233 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.233 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.234 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:05:31.241 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.241 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.242 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.247 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.253 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:05:31.257 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.257 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.258 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.263 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.263 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.263 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.274 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.274 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.275 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.275 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.275 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.275 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.286 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.287 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.287 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:05:31.289 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.289 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.289 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.296 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.296 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.297 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:05:31.308 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.308 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.309 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.309 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.309 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.309 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:05:31.319 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:05:31.321 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.321 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.322 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:05:31.330 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:05:31.345 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.346 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.346 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.346 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.346 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.347 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.347 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.347 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.347 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.347 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.348 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.348 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.349 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.352 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.352 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.352 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.361 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.361 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.361 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.362 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.362 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.362 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.368 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.368 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.369 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.369 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.369 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.373 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.374 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.374 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.375 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.375 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.376 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.376 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.376 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.377 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.377 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.377 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.390 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.390 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.391 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.391 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.391 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.403 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.404 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.404 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.404 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.409 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.409 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.410 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.411 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.411 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.411 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.411 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.412 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.412 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.412 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.413 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.422 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.423 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.423 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.423 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.423 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.427 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.427 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.427 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.428 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.432 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.432 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.433 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.434 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:05:31.437 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:05:31.441 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.441 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.441 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.442 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.442 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.443 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.443 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.444 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:05:31.454 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.455 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.455 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.455 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.455 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.456 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.456 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.456 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:05:31.456 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:05:31.459 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:05:31.473 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.474 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.474 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.475 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.475 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:31.475 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.475 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.476 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.476 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.476 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:31.478 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.478 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.478 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.480 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.481 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.481 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.486 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.488 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:05:31.488 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.488 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.488 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:05:31.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,561b2a470b734080,3dba944ef50c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=561b2a470b734080, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:31.493 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,561b2a470b734080,3dba944ef50c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=561b2a470b734080, duration=5 ms] 2023-05-15 14:05:31.493 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.501 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:05:31.502 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:31.502 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.503 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:05:31.503 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.504 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:05:31.505 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.505 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.505 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:05:31.514 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.516 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.517 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:05:31.517 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:31.517 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:31.518 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:05:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:05:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:05:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:05:31.525 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,4ee9ef5d4501] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131901, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:31.527 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:05:31.529 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:05:31.529 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:05:31.529 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:05:31.534 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,80aacbc94330] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131900, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:05:31.667 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116299, status:SUCCESS 2023-05-15 14:05:31.668 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116299, status:SUCCESS 2023-05-15 14:05:31.677 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116300, status:SUCCESS 2023-05-15 14:05:31.678 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116300, status:SUCCESS 2023-05-15 14:05:31.849 INFO 11 --- [pool-ocp-schedules-11,,2d2c753d2221] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:31.850 INFO 11 --- [pool-ocp-schedules-11,,2d2c753d2221] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:32.009 INFO 11 --- [pool-ocp-schedules-9,,8b73a206fece] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:05:32.012 INFO 11 --- [pool-ocp-schedules-9,,8b73a206fece] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:05:32.033 INFO 11 --- [pool-ocp-schedules-4,,69c280fe6002] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:05:32.035 INFO 11 --- [pool-ocp-schedules-4,,69c280fe6002] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:05:32.036 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.036 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.036 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.037 INFO 11 --- [pool-ocp-schedules-7,,b83b8a3e06f3] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:05:32.037 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.038 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.038 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.038 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.039 INFO 11 --- [pool-ocp-schedules-7,,b83b8a3e06f3] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:05:32.039 INFO 11 --- [ForkJoinPool-66033-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:05:32.044 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.044 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.044 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.045 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.050 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.050 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.050 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.054 INFO 11 --- [ForkJoinPool-66033-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:05:32.071 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.072 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.072 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.072 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.074 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.074 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.074 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:05:32.085 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.086 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.086 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.086 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.090 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.090 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.091 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:05:32.117 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:32.118 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.118 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.118 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.119 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.120 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.120 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.120 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:05:32.122 WARN 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:05:32.122 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.122 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.123 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:32.124 INFO 11 --- [ForkJoinPool-66035-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:05:32.124 INFO 11 --- [ForkJoinPool-66035-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:05:32.124 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:05:32.135 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.135 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:05:32.147 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.157 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:32.158 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.158 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.159 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.160 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.161 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:05:32.161 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.161 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.169 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:05:32.169 INFO 11 --- [ForkJoinPool-66035-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130136321000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:05:32.172 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.172 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.173 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:05:32.177 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.177 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.177 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:05:32.180 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:05:32.196 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.196 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:05:32.210 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.210 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:05:32.221 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.221 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:05:32.233 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.233 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:05:32.244 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.245 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:05:32.263 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:05:32.284 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:05:32.285 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.285 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.285 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.286 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.291 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:05:32.291 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:05:32.291 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:05:32.316 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:05:32.316 INFO 11 --- [ForkJoinPool-66035-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130146903000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:05:32.768 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:32.769 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:34.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:34.830936+08:00, lastProcessTime=2023-05-15T14:05:24.829232+08:00 2023-05-15 14:05:34.860 INFO 11 --- [alarm-detect-15,,20636c725815] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130714 ~ 1684130724 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=109.0, values={1684130720=109.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:05:34.862 INFO 11 --- [alarm-detect-15,,20636c725815] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, currentState=Pending 2023-05-15 14:05:34.866 INFO 11 --- [alarm-detect-3,,f2c78bd3bb21] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130714 ~ 1684130724 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130720=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:34.870 INFO 11 --- [alarm-detect-15,,602aa9a94ff3] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130664 ~ 1684130724 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389097.938, values={1684130700=389097.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:05:34.871 INFO 11 --- [alarm-detect-15,,602aa9a94ff3] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130664 ~ 1684130724 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993895.107, values={1684130700=993895.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:34.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:34.830936+08:00 2023-05-15 14:05:35.769 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:35.770 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:35.990 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:35.993 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:35.997 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:36.006 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:36.127 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sessions?page=1&size=10&activeOnly=false, client=192.168.80.229, traceId=4724b297730e4e4b, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.listTenantSessions(Pageable,Long,Long,String,String,String,Boolean), args=Page request [number: 0, size 10, sort: id: DESC],2,6,false,] 2023-05-15 14:05:36.153 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:05:36.153 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:05:36.165 ERROR 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573868) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573868) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:36.166 ERROR 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] com.alibaba.druid.pool.DruidDataSource : {dataSource-95} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573868) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573868) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:36.167 INFO 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] com.alibaba.druid.pool.DruidDataSource : {dataSource-95,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:05:36.167 INFO 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] com.alibaba.druid.pool.DruidDataSource : {dataSource-95} closing ... 2023-05-15 14:05:36.167 INFO 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] com.alibaba.druid.pool.DruidDataSource : {dataSource-95} closed 2023-05-15 14:05:36.167 ERROR 11 --- [obsdk-init-thread-12,4724b297730e4e4b,a7672e4e095e] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573868) Get Location Cache Fail 2023-05-15 14:05:36.174 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573868) Get Location Cache Fail 2023-05-15 14:05:36.175 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573868) Get Location Cache Fail 2023-05-15 14:05:36.175 WARN 11 --- [http-nio-0.0.0.0-8080-exec-63,4724b297730e4e4b,960aa39fd0e3] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/sessions client=192.168.80.229, traceId=4724b297730e4e4b, duration=50 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:05:36.183 ERROR 11 --- [Druid-ConnectionPool-Create-484461599,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573871) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573871) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:05:36.319 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,41ceb8fa91ae49c9,f45e8b84019a] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&version=2&ObClusterId=1, client=10.xxx.xx.10, traceId=41ceb8fa91ae49c9, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=ocpcluster,1,2,{"ObRegion":"ocpcluster","ObCluster":"ocpcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684130736316210,"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 14:05:36.323 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,41ceb8fa91ae49c9,f45e8b84019a] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=ocpcluster, clusterId=1, affectRows=1 2023-05-15 14:05:36.331 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,41ceb8fa91ae49c9,f45e8b84019a] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.10, traceId=41ceb8fa91ae49c9, duration=13 ms] 2023-05-15 14:05:36.975 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:38.178 INFO 11 --- [pool-ocp-schedules-11,,64dcc5c097b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:38.263 INFO 11 --- [pool-overview-container-7,,3ec82e387fe1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:38.512 INFO 11 --- [pool-scanner--15972641-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:38.770 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:38.775 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:41.776 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:41.777 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:42.385 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:42.976 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,d0a1324e1e734bfd,298c15dc4d6f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=d0a1324e1e734bfd, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:42.982 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,d0a1324e1e734bfd,298c15dc4d6f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=d0a1324e1e734bfd, duration=7 ms] 2023-05-15 14:05:44.777 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:44.778 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:44.830881+08:00, lastProcessTime=2023-05-15T14:05:34.830936+08:00 2023-05-15 14:05:44.851 INFO 11 --- [alarm-detect-7,,269900323528] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130714 ~ 1684130724 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=109.0, values={1684130720=109.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:05:44.854 INFO 11 --- [alarm-detect-3,,65838aa27828] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130724 ~ 1684130734 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130730=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:44.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:44.830881+08:00 2023-05-15 14:05:45.483 INFO 11 --- [pool-ocp-schedules-12,,296e350b63fb] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:05:45.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,54d7b10dc18646e1,72805d83b3df] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=54d7b10dc18646e1, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:45.805 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,54d7b10dc18646e1,72805d83b3df] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=54d7b10dc18646e1, duration=3 ms] 2023-05-15 14:05:45.993 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:45.995 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:45.999 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:46.008 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:46.595 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/parameters?null, client=192.168.80.229, traceId=3d60acd0a30a4a2b, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantParameterController.listTenantParameters(Long,Long,Boolean,Boolean), args=2,6,] 2023-05-15 14:05:46.624 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:05:46.624 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:05:46.642 ERROR 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573874) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573874) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:46.644 ERROR 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] com.alibaba.druid.pool.DruidDataSource : {dataSource-96} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573874) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573874) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:05:46.644 INFO 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] com.alibaba.druid.pool.DruidDataSource : {dataSource-96,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:05:46.644 INFO 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] com.alibaba.druid.pool.DruidDataSource : {dataSource-96} closing ... 2023-05-15 14:05:46.644 INFO 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] com.alibaba.druid.pool.DruidDataSource : {dataSource-96} closed 2023-05-15 14:05:46.648 ERROR 11 --- [obsdk-init-thread-13,3d60acd0a30a4a2b,b7e5d3ca6fcd] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573874) Get Location Cache Fail 2023-05-15 14:05:46.649 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573874) Get Location Cache Fail 2023-05-15 14:05:46.649 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573874) Get Location Cache Fail 2023-05-15 14:05:46.649 WARN 11 --- [http-nio-0.0.0.0-8080-exec-62,3d60acd0a30a4a2b,7e1703e643cb] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/parameters client=192.168.80.229, traceId=3d60acd0a30a4a2b, duration=56 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:05:46.656 ERROR 11 --- [Druid-ConnectionPool-Create-1347242613,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573877) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573877) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:05:46.903 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6058d82842cc4dd9,5dd8f0324de7] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&version=2&ObClusterId=1, client=10.xxx.xx.11, traceId=6058d82842cc4dd9, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obcluster,1,2,{"ObRegion":"obcluster","ObCluster":"obcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684130746900468,"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 14:05:46.907 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6058d82842cc4dd9,5dd8f0324de7] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=obcluster, clusterId=1, affectRows=1 2023-05-15 14:05:46.908 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6058d82842cc4dd9,5dd8f0324de7] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.11, traceId=6058d82842cc4dd9, duration=6 ms] 2023-05-15 14:05:46.978 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:47.778 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:47.779 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:48.200 INFO 11 --- [pool-ocp-schedules-12,,7e2cacced67d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:48.264 INFO 11 --- [pool-overview-container-1,,8d17b27ae468] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:48.512 INFO 11 --- [pool-scanner--15972741-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:50.779 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:50.781 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:51.815 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:51.817 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:51.819 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:51.855 INFO 11 --- [pool-ocp-schedules-16,,5034629645f4] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:51.855 INFO 11 --- [pool-ocp-schedules-16,,5034629645f4] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:05:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,656fa702b2fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40050} 2023-05-15 14:05:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,656fa702b2fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:05:53.781 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:53.782 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:05:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:05:54.831720+08:00, lastProcessTime=2023-05-15T14:05:44.830881+08:00 2023-05-15 14:05:54.887 INFO 11 --- [alarm-detect-14,,6a6029e8a0e5] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130714 ~ 1684130724 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=109.0, values={1684130720=109.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:05:54.899 INFO 11 --- [alarm-detect-13,,828e4e6543d1] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130734 ~ 1684130744 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130740=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:05:54.904 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:05:54.831720+08:00 2023-05-15 14:05:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,197a1deddcbf] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:05:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,23fb38ecfb31] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:05:55.995 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:55.998 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:56.001 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:56.010 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:56.782 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:56.783 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:05:56.981 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:57.388 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:05:58.222 INFO 11 --- [pool-ocp-schedules-9,,17718371d099] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:05:58.264 INFO 11 --- [pool-overview-container-6,,e203282a6f78] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:05:58.515 INFO 11 --- [pool-scanner--15972835-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:05:58.601 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,d5a01f4585344c4f,dbadb3f6f685] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=d5a01f4585344c4f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:58.608 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,d5a01f4585344c4f,dbadb3f6f685] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=d5a01f4585344c4f, duration=8 ms] 2023-05-15 14:05:59.074 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,9d1975b370e945d6,13ed3d121e5d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=9d1975b370e945d6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:05:59.081 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,9d1975b370e945d6,13ed3d121e5d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=9d1975b370e945d6, duration=8 ms] 2023-05-15 14:05:59.784 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:05:59.784 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130760 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-19,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-19,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:06:00.000 INFO 11 --- [pool-ocp-schedules-5,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:06:00.013 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:06:00.013 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:06:00.013 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:06:00.013 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:06:00.020 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:06:00.020 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:06:00.020 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:06:00.020 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:06:00.048 INFO 11 --- [pool-ocp-schedules-19,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:06:00.059 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:06:00.067 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:06:00.067 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:06:00.067 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:06:00.067 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:06:00.067 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:06:00.075 WARN 11 --- [pool-ocp-schedules-5,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:06:00.075 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:06:00.571 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:06:00.832 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:04:01.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.832 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:04:01.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.841 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:06:00.841 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.841 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.841 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:00.855 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.856 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:06:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.860 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:00.865 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.866 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:06:00.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:00.879 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.879 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.888 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:06:00.888 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.888 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:00.892 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:31.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.893 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:05:31.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.897 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:06:00.897 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.897 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:00.905 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:06:00.0, currentDate=Mon May 15 14:06:00 CST 2023 2023-05-15 14:06:00.905 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:06:00.0} 2023-05-15 14:06:00.908 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:06:00.908 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:00.909 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:01.033 WARN 11 --- [pool-ocp-schedules-5,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:06:01.034 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.014281169S 2023-05-15 14:06:01.035 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:06:01.156 INFO 11 --- [pool-ocp-schedules-5,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:06:01.233 INFO 11 --- [pool-ocp-schedules-5,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:06:01.277 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:06:01.277 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:06:01.277 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:06:01.277 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:06:01.277 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:06:01.278 INFO 11 --- [pool-ocp-schedules-5,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:06:01.287 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:06:01.431 INFO 11 --- [pool-ocp-schedules-5,,] o.s.boot.SpringApplication : Started application in 1.418 seconds (JVM running for 600417.404) 2023-05-15 14:06:01.431 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.418511006S 2023-05-15 14:06:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,cbbf9d4c936a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130820, interval=60 2023-05-15 14:06:01.529 INFO 11 --- [pool-prometheus-minute-schedule-2,,7c63ef956995] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130820, interval=60 2023-05-15 14:06:01.546 INFO 11 --- [pool-prometheus-minute-schedule-1,,cbbf9d4c936a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:06:01.546 INFO 11 --- [pool-prometheus-minute-schedule-1,,cbbf9d4c936a] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130761, count=8, elapsed=1682446630785 2023-05-15 14:06:01.547 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,985782562f8c4c09,98f9211b2531] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=985782562f8c4c09, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:06:01.549 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,985782562f8c4c09,98f9211b2531] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=985782562f8c4c09, duration=3 ms] 2023-05-15 14:06:01.555 INFO 11 --- [pool-prometheus-minute-schedule-2,,7c63ef956995] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:06:01.555 INFO 11 --- [pool-prometheus-minute-schedule-2,,7c63ef956995] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130761, count=6, elapsed=1682446630794 2023-05-15 14:06:01.625 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131906, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.627 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131906, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:06:01.615+08:00, endTime=null} 2023-05-15 14:06:01.636 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131907, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.637 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131907, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:06:01.633+08:00, endTime=null} 2023-05-15 14:06:01.650 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131904, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.652 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131904, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:06:01.647+08:00, endTime=null} 2023-05-15 14:06:01.653 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c595ea82f64f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131906, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116303, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.615+08:00, sub_task_instance_id=131906}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfd427643e32] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131907, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116304, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.633+08:00, sub_task_instance_id=131907}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.664 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131905, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.665 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:06:01.666 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:06:01.666 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:06:01.666 INFO 11 --- [pool-ocp-schedules-5,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:06:01.666 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131905, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:06:01.662+08:00, endTime=null} 2023-05-15 14:06:01.677 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c595ea82f64f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:06:01.677 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,fefd28baa157] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131904, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116301, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.647+08:00, sub_task_instance_id=131904}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.677 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131908, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.678 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c595ea82f64f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:06:01.681 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131908, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:06:01.675+08:00, endTime=null} 2023-05-15 14:06:01.684 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfd427643e32] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:06:01.684 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfd427643e32] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:06:01.685 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,e45cafa88607] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131905, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116302, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.662+08:00, sub_task_instance_id=131905}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.688 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfd427643e32] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:06:01.694 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131909, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.695 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:06:01.695 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:06:01.695 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:06:01.695 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:06:01.695 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:06:01.700 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,fefd28baa157] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131904, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.700 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,e45cafa88607] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131905, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.700 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c595ea82f64f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131906, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.701 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:06:01.701 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,dfd427643e32] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131907, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.701 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:06:01.702 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:06:01.702 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:06:01.702 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:06:01.702 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131909, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:06:01.692+08:00, endTime=null} 2023-05-15 14:06:01.728 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131903, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:01.731 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131903, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:06:01.723+08:00, endTime=null} 2023-05-15 14:06:01.734 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d4445ab9c1c7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131908, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116305, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.675+08:00, sub_task_instance_id=131908}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.734 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d4445ab9c1c7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:06:01.738 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:06:01.738 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ff2b118c13f8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131909, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116306, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.692+08:00, sub_task_instance_id=131909}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.739 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:06:01.739 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:06:01.744 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:06:01.744 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:06:01.747 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:06:01.756 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:06:01.756 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:06:01.756 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:06:01.756 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:06:01.756 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ff2b118c13f8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131909, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.756 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:06:01.757 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:06:01.757 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:06:01.760 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:06:01.765 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:06:01.765 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4ca7f83d14cc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131903, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116301, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:01.721+08:00, sub_task_instance_id=131903}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:06:01.771 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.772 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.772 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.773 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.773 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.774 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.775 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.778 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:06:01.779 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:06:01.790 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.790 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.791 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:06:01.791 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:06:01.792 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.792 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.793 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.793 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.793 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.793 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.793 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.794 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:06:01.794 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.794 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.795 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.795 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:06:01.796 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.796 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.797 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:06:01.797 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.797 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:06:01.798 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.798 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.798 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:01.798 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.798 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:06:01.799 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:06:01.801 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:06:01.801 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4ca7f83d14cc] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131903, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:01.802 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:06:01.803 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.803 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.804 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:06:01.804 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:01.808 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:01.808 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:01.809 INFO 11 --- [pool-3244954-thread-2,fb27d19334114ed7,bf904d0f915c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:06:01.809 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:06:01.809 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:06:01.810 INFO 11 --- [pool-3244954-thread-6,fb27d19334114ed7,4cd6f92353cd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:06:01.810 INFO 11 --- [pool-3244954-thread-3,fb27d19334114ed7,e46c54caa8b9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:06:01.811 INFO 11 --- [pool-3244954-thread-4,fb27d19334114ed7,d585eec8f709] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:06:01.813 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:06:01.813 INFO 11 --- [pool-3244954-thread-5,fb27d19334114ed7,3e8a0cb43b4c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:06:01.815 INFO 11 --- [pool-3244954-thread-1,fb27d19334114ed7,3cdf3ca517d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:06:01.824 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d4445ab9c1c7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:06:01.831 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d4445ab9c1c7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131908, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:02.054 INFO 11 --- [pool-ocp-schedules-5,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:06:02.260 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116301, status:SUCCESS 2023-05-15 14:06:02.261 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116301, status:SUCCESS 2023-05-15 14:06:02.271 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116302, status:SUCCESS 2023-05-15 14:06:02.272 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116302, status:SUCCESS 2023-05-15 14:06:02.283 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116303, status:SUCCESS 2023-05-15 14:06:02.284 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116303, status:SUCCESS 2023-05-15 14:06:02.294 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116304, status:SUCCESS 2023-05-15 14:06:02.296 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116304, status:SUCCESS 2023-05-15 14:06:02.307 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116305, status:SUCCESS 2023-05-15 14:06:02.308 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116305, status:SUCCESS 2023-05-15 14:06:02.318 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116306, status:SUCCESS 2023-05-15 14:06:02.320 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116306, status:SUCCESS 2023-05-15 14:06:02.785 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:02.786 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:04.714 INFO 11 --- [pool-ocp-schedules-4,,793e10abad09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:06:04.721 INFO 11 --- [pool-ocp-schedules-4,,793e10abad09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:06:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:04.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:04.831101+08:00, lastProcessTime=2023-05-15T14:05:54.831720+08:00 2023-05-15 14:06:04.859 INFO 11 --- [alarm-detect-4,,71c4d03e56f8] c.o.o.s.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, activeAt=2023-05-15T14:05:34.849+08:00 2023-05-15 14:06:04.859 INFO 11 --- [alarm-detect-4,,71c4d03e56f8] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType tenant_cpu_percent_over_threshold targets.size 10 was marked as not match 2023-05-15 14:06:04.864 INFO 11 --- [alarm-detect-13,,f81c4d8fbdf8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130744 ~ 1684130754 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.903, values={1684130750=245.903}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:04.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:04.831101+08:00 2023-05-15 14:06:05.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:06.0, nextRun=2023-05-15 14:06:05.0, currentDate=Mon May 15 14:06:05 CST 2023 2023-05-15 14:06:05.043 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:05:06.0, nextRunTime=2023-05-15 14:06:05.0} 2023-05-15 14:06:05.047 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:06:05.047 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:05.047 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:05.406 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,21516d222d6c4e31,1627adbccc36] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=21516d222d6c4e31, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:05.411 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,21516d222d6c4e31,1627adbccc36] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=21516d222d6c4e31, duration=5 ms] 2023-05-15 14:06:05.748 INFO 11 --- [pool-ocp-schedules-2,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-05-15 14:06:05.786 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:05.787 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:05.808 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131910, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:05.810 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131910, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:06:05.806+08:00, endTime=null} 2023-05-15 14:06:05.829 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a7c3e290b50d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131910, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116307, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:05.805+08:00, sub_task_instance_id=131910}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:05.851 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,dfb7286e09cf] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:06:05.852 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,d0019fc5fb5a] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:06:05.853 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,11d65933190e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:06:05.898 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,dfb7286e09cf] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:06:05.899 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,11d65933190e] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:06:05.904 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,d0019fc5fb5a] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:06:05.921 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,a7c3e290b50d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131910, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:05.998 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:06.000 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:06.003 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:06.013 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:06.409 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116307, status:SUCCESS 2023-05-15 14:06:06.410 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116307, status:SUCCESS 2023-05-15 14:06:06.985 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:08.242 INFO 11 --- [pool-ocp-schedules-4,,415179087bee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:08.266 INFO 11 --- [pool-overview-container-2,,760da2f45cc4] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:06:08.266 INFO 11 --- [pool-overview-container-2,,760da2f45cc4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:08.523 INFO 11 --- [pool-scanner--15972928-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:06:08.787 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:08.791 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:10.300 INFO 11 --- [pool-ocp-schedules-17,,e072c96f4452] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:06:11.792 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:11.793 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:11.860 INFO 11 --- [pool-ocp-schedules-11,,269d1dde0f17] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:11.861 INFO 11 --- [pool-ocp-schedules-11,,269d1dde0f17] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:12.390 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:14.417 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130700, nodeCount=90, affectRows=90 2023-05-15 14:06:14.418 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130700, nodeCount=90, validCount=90 2023-05-15 14:06:14.468 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130700, nodeCount=905, affectRows=905 2023-05-15 14:06:14.476 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130700, nodeCount=905, validCount=905 2023-05-15 14:06:14.478 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130700, nodeCount=842, affectRows=842 2023-05-15 14:06:14.485 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130700, nodeCount=842, validCount=842 2023-05-15 14:06:14.488 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130700, nodeCount=853, affectRows=853 2023-05-15 14:06:14.495 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130700, nodeCount=853, validCount=853 2023-05-15 14:06:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:14.793 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:14.794 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:14.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:14.830683+08:00, lastProcessTime=2023-05-15T14:06:04.831101+08:00 2023-05-15 14:06:14.857 INFO 11 --- [alarm-detect-6,,b36e16ae5bc0] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130754 ~ 1684130764 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130760=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:14.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:14.830683+08:00 2023-05-15 14:06:16.000 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:16.003 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:16.005 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:16.015 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:16.388 INFO 11 --- [pool-ocp-schedules-2,,a0b0f2185928] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:06:16.389 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:06:16.389 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:06:16.389 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:06:16.389 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:06:16.389 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:06:16.395 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:06:16.395 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:06:16.396 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:06:16.396 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:06:16.396 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:06:16.420 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:06:16.421 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:06:16.422 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:06:16.422 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:06:16.422 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:06:16.422450535+08:00 2023-05-15 14:06:16.423 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:06:16.422+08:00 2023-05-15 14:06:16.423 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:06:16.423 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:06:16.423325326+08:00 2023-05-15 14:06:16.423 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:06:16.423+08:00 2023-05-15 14:06:16.424 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:06:16.423770758+08:00 2023-05-15 14:06:16.424 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:06:16.424+08:00 2023-05-15 14:06:16.424 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:06:16.424563767+08:00 2023-05-15 14:06:16.425 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:06:16.424+08:00 2023-05-15 14:06:16.425 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:06:16.424845589+08:00 2023-05-15 14:06:16.425 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:06:16.425+08:00 2023-05-15 14:06:16.433 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:06:16.422+08:00 and status AVAILABLE 2023-05-15 14:06:16.436 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:06:16.424+08:00 and status AVAILABLE 2023-05-15 14:06:16.439 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:06:16.440 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:06:16.425+08:00 and status AVAILABLE 2023-05-15 14:06:16.442 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:06:16.424+08:00 and status AVAILABLE 2023-05-15 14:06:16.442 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:06:16.459 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130776 0 0 , error= 2023-05-15 14:06:16.460 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:06:16.459+08:00 2023-05-15 14:06:16.460 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:06:16.423+08:00 and status AVAILABLE 2023-05-15 14:06:16.474 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130776 0 0 , error= 2023-05-15 14:06:16.475 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:06:16.474+08:00 2023-05-15 14:06:16.475 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:06:16.459+08:00 and status AVAILABLE 2023-05-15 14:06:16.476 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:06:16.476 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:06:16.478 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:06:16.495 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:06:16.474+08:00 and status AVAILABLE 2023-05-15 14:06:16.500 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:06:16.500 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130776 0 0 , error= 2023-05-15 14:06:16.507 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130776 0 0 , error= 2023-05-15 14:06:16.508 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:06:16.508+08:00 2023-05-15 14:06:16.508 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:06:16.508+08:00 2023-05-15 14:06:16.508 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:06:16.518 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130776 0 0 , error= 2023-05-15 14:06:16.520 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:06:16.520+08:00 2023-05-15 14:06:16.520 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:06:16.508+08:00 and status AVAILABLE 2023-05-15 14:06:16.520 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:06:16.508+08:00 and status AVAILABLE 2023-05-15 14:06:16.523 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:06:16.523 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:06:16.525 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:06:16.520+08:00 and status AVAILABLE 2023-05-15 14:06:16.526 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:06:16.528 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:06:16.529 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:06:16.536 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:06:16.536 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:06:16.540 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:06:16.554 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:06:16.556 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:06:16.564 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:06:16.564 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:06:16.566 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:06:16.602 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:06:16.609 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:06:16.615 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:06:16.623 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:06:16.626 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:06:16.630 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:06:16.639 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:06:16.642 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:06:16.646 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:06:16.648 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:06:16.656 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:06:16.657 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:06:16.659 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:06:16.666 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:06:16.683 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:06:16.686 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:06:16.700 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:06:16.703 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:06:16.704 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:06:16.712 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:06:16.719 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:06:16.720 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:06:16.721 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:06:16.725 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:06:16.736 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:06:16.737 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:06:16.740 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:06:16.762 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:06:16.765 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:06:16.778 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:06:16.778 INFO 11 --- [pool-Check host status21,,aef0f0ff0019] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:06:16.790 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:06:16.792 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:06:16.795 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:06:16.796 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:06:16.801 INFO 11 --- [pool-Check host status29,,424a089c5493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:06:16.803 INFO 11 --- [pool-Check host status4,,6b243e8da35b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:06:16.805 INFO 11 --- [pool-Check host status7,,979719116da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:06:16.872 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:06:16.886 INFO 11 --- [pool-Check host status6,,a16fc0afd2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:06:16.909 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:06:16.909 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:06:16.909 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:06:16.909 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:06:16.909 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:06:16.916 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:06:16.916 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:06:16.916 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:06:16.916 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:06:16.916 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:06:16.944 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:06:16.945 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:06:16.946 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:06:16.947 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:06:16.949 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status27,,827dd93c0397] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:06:16.965 INFO 11 --- [pool-Check host status11,,3d82d7576d05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status20,,e2b2d030bb94] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:06:16.968 INFO 11 --- [pool-Check host status5,,4c78225aaf5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:06:16.970 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:06:16.971 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:06:16.971 INFO 11 --- [pool-Check host status23,,3e90eef8ce71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:06:16.988 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:17.083 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,eef5fb3878a9440d,3a9b797ca506] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=eef5fb3878a9440d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:17.089 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,eef5fb3878a9440d,3a9b797ca506] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=eef5fb3878a9440d, duration=6 ms] 2023-05-15 14:06:17.794 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:17.795 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:18.264 INFO 11 --- [pool-overview-container-4,,b30bf8f99e39] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:18.265 INFO 11 --- [pool-ocp-schedules-3,,6b0bf6d72a25] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:18.513 INFO 11 --- [pool-scanner--15973028-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:19.696 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,b77fac0a13e84392,fe2189133c4f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=b77fac0a13e84392, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:19.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,b77fac0a13e84392,fe2189133c4f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=b77fac0a13e84392, duration=6 ms] 2023-05-15 14:06:20.498 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:21.0, nextRun=2023-05-15 14:06:20.0, currentDate=Mon May 15 14:06:20 CST 2023 2023-05-15 14:06:20.498 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:05:21.0, nextRunTime=2023-05-15 14:06:20.0} 2023-05-15 14:06:20.502 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:06:20.502 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:20.502 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:20.795 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:20.797 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:21.049 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131911, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:21.052 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131911, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:06:21.046+08:00, endTime=null} 2023-05-15 14:06:21.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,af63137b79e3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131911, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116308, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:21.046+08:00, sub_task_instance_id=131911}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:21.082 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,af63137b79e3] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:06:21.083 INFO 11 --- [ForkJoinPool-66038-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:06:21.083 INFO 11 --- [ForkJoinPool-66038-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:06:21.089 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,af63137b79e3] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131911, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:21.648 INFO 11 --- [pool-scanner--15973029-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:21.671 INFO 11 --- [pool-scanner--15973030-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:21.680 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116308, status:SUCCESS 2023-05-15 14:06:21.682 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116308, status:SUCCESS 2023-05-15 14:06:21.697 INFO 11 --- [pool-scanner--15973031-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:21.718 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.722 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.762 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.766 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.780 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.783 INFO 11 --- [pool-ocp-schedules-6,,402840ef97cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:21.817 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:21.819 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:21.820 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:23.797 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:23.798 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:24.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:24.833711+08:00, lastProcessTime=2023-05-15T14:06:14.830683+08:00 2023-05-15 14:06:24.991 INFO 11 --- [alarm-detect-2,,9c804c6914be] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130764 ~ 1684130774 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130770=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:25.007 INFO 11 --- [alarm-detect-9,,1dac1869835a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130714 ~ 1684130774 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389157.988, values={1684130760=389157.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:06:25.008 INFO 11 --- [alarm-detect-9,,1dac1869835a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130714 ~ 1684130774 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=993955.358, values={1684130760=993955.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:25.043 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:24.833711+08:00 2023-05-15 14:06:25.401 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/performance/config?id=2&tenantId=6, client=192.168.80.229, traceId=629a929fdb894f60, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getClusterPerfConfig(Long,Long), args=2,6,] 2023-05-15 14:06:25.401 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,72f9480189c94c38,bb9e9feed6a1] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch?id=2&tenantId=6, client=192.168.80.229, traceId=72f9480189c94c38, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitch(Long,Long), args=2,6,] 2023-05-15 14:06:25.403 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,3f262baa99d942ae,d08650d9b7a9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events?null, client=192.168.80.229, traceId=3f262baa99d942ae, method=PaginatedResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchEvents(Long,Long,OffsetDateTime,OffsetDateTime,Pageable), args=2,6,Page request [number: 0, size 2000, sort: timestamp: ASC],] 2023-05-15 14:06:25.419 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,3f262baa99d942ae,d08650d9b7a9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events client=192.168.80.229, traceId=3f262baa99d942ae, duration=18 ms] 2023-05-15 14:06:25.433 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.434 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:25.437 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,72f9480189c94c38,bb9e9feed6a1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch client=192.168.80.229, traceId=72f9480189c94c38, duration=38 ms] 2023-05-15 14:06:25.468 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/outlines?null, client=192.168.80.229, traceId=9a127d5f2da84e34, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObOutlineController.getSqlOutline(Long,Long,String,String,OffsetDateTime,OffsetDateTime,Boolean), args=2,6,] 2023-05-15 14:06:25.475 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.475 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.476 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() REGEXP 'OceanBase.*?CE' 2023-05-15 14:06:25.487 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.487 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.488 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2023-05-15 14:06:25.493 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:06:25.493 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:06:25.504 ERROR 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573881) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573881) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:25.506 ERROR 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] com.alibaba.druid.pool.DruidDataSource : {dataSource-97} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573881) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573881) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:25.506 INFO 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] com.alibaba.druid.pool.DruidDataSource : {dataSource-97,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:06:25.507 INFO 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] com.alibaba.druid.pool.DruidDataSource : {dataSource-97} closing ... 2023-05-15 14:06:25.510 INFO 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] com.alibaba.druid.pool.DruidDataSource : {dataSource-97} closed 2023-05-15 14:06:25.510 ERROR 11 --- [obsdk-init-thread-14,9a127d5f2da84e34,8af211cef230] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573881) Get Location Cache Fail 2023-05-15 14:06:25.511 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573881) Get Location Cache Fail 2023-05-15 14:06:25.511 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573881) Get Location Cache Fail 2023-05-15 14:06:25.511 WARN 11 --- [http-nio-0.0.0.0-8080-exec-52,9a127d5f2da84e34,f864a0d4a5cc] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/outlines client=192.168.80.229, traceId=9a127d5f2da84e34, duration=44 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:06:25.519 ERROR 11 --- [Druid-ConnectionPool-Create-543102066,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573884) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573884) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:06:25.580 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,6e3acda93f894eda,e18852e3aeaf] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents?startTime=2023-05-15T13%3A36%3A25%2B08%3A00&endTime=2023-05-15T14%3A06%3A25%2B08%3A00, client=192.168.80.229, traceId=6e3acda93f894eda, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchOffIntervalLimit3(Long,Long,OffsetDateTime,OffsetDateTime), args=2,6,2023-05-15T13:36:25+08:00,2023-05-15T14:06:25+08:00,] 2023-05-15 14:06:25.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A36%3A25%2B08%3A00&endTime=2023-05-15T14%3A06%3A25%2B08%3A00&inner=false, client=192.168.80.229, traceId=4cbf20f0c3cd4dad, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:36:25+08:00,2023-05-15T14:06:25+08:00,false,] 2023-05-15 14:06:25.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.597 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,6e3acda93f894eda,e18852e3aeaf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents client=192.168.80.229, traceId=6e3acda93f894eda, duration=18 ms] 2023-05-15 14:06:25.610 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.610 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.610 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.611 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:25.687 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.688 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.688 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:25.690 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.690 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.690 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2023-05-15 14:06:25.691 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:06:25.695 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:26.0, nextRun=2023-05-15 14:06:25.0, currentDate=Mon May 15 14:06:25 CST 2023 2023-05-15 14:06:25.695 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:05:26.0, nextRunTime=2023-05-15 14:06:25.0} 2023-05-15 14:06:25.706 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:06:25.706 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:25.706 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:25.721 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}} 2023-05-15 14:06:25.771 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.771 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.772 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:06:25.772 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:06:25.780 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:25.781 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.782 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.782 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.783 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:25.873 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.873 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.874 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:25.885 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.899 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,4cbf20f0c3cd4dad,0eb13ed0af1c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=4cbf20f0c3cd4dad, duration=318 ms] 2023-05-15 14:06:25.932 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A36%3A25%2B08%3A00&endTime=2023-05-15T14%3A06%3A25%2B08%3A00&inner=false, client=192.168.80.229, traceId=e56a90e279754bcf, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:36:25+08:00,2023-05-15T14:06:25+08:00,false,] 2023-05-15 14:06:25.933 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.933 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:25.971 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.972 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.972 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.973 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:25.978 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:25.978 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:25.979 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:25.983 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:06:25.986 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}} 2023-05-15 14:06:26.003 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:26.006 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:26.009 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:26.017 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:26.038 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:06:26.041 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.041 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.054 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e56a90e279754bcf,892eee6d7257] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=e56a90e279754bcf, duration=123 ms] 2023-05-15 14:06:26.084 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A36%3A25%2B08%3A00&endTime=2023-05-15T14%3A06%3A25%2B08%3A00&inner=false, client=192.168.80.229, traceId=40713c102fc344a4, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:36:25+08:00,2023-05-15T14:06:25+08:00,false,] 2023-05-15 14:06:26.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.097 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,629a929fdb894f60,d2b3ed5fc363] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/performance/config client=192.168.80.229, traceId=629a929fdb894f60, duration=698 ms] 2023-05-15 14:06:26.107 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:26.108 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:26.108 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:26.109 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:26.113 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:26.113 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:26.114 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:26.117 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:06:26.119 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:36:25, end=2023-05-15T14:06:25}} 2023-05-15 14:06:26.126 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131912, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:26.128 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131912, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:06:26.123+08:00, endTime=null} 2023-05-15 14:06:26.153 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131912, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116309, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:26.122+08:00, sub_task_instance_id=131912}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:06:26.177 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:06:26.181 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.181 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:06:26.198 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,40713c102fc344a4,424eede1102f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=40713c102fc344a4, duration=116 ms] 2023-05-15 14:06:26.220 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:06:26.220 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:06:26.221 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:06:26.222 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:06:26.222 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:06:26.228 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,f3ff9686437b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131912, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:26.779 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116309, status:SUCCESS 2023-05-15 14:06:26.781 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116309, status:SUCCESS 2023-05-15 14:06:26.798 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:26.799 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:26.991 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:27.130 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/63577C163A26A8D3FC2EA6A2C92C4E83/text?startTime=2023-05-15T13%3A36%3A25%2B08%3A00&endTime=2023-05-15T14%3A06%3A25%2B08%3A00, client=192.168.80.229, traceId=1acfa4ab42c549c3, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,63577C163A26A8D3FC2EA6A2C92C4E83,2023-05-15T13:36:25+08:00,2023-05-15T14:06:25+08:00,] 2023-05-15 14:06:27.160 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.161 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.161 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:27.162 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:27.173 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.173 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:27.174 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:27.187 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:27.189 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.189 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.190 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:27.191 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:27.230 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:27.230 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:27.234 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:06:27.393 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:27.410 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1acfa4ab42c549c3,62d173ba67e6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/63577C163A26A8D3FC2EA6A2C92C4E83/text client=192.168.80.229, traceId=1acfa4ab42c549c3, duration=282 ms] 2023-05-15 14:06:28.264 INFO 11 --- [pool-overview-container-8,,4791bc68969f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:28.285 INFO 11 --- [pool-ocp-schedules-16,,7de899cf435e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:28.513 INFO 11 --- [pool-scanner--15973129-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:29.799 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:29.800 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:30.857 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:31.0, nextRun=2023-05-15 14:06:30.0, currentDate=Mon May 15 14:06:30 CST 2023 2023-05-15 14:06:30.858 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:05:31.0, nextRunTime=2023-05-15 14:06:30.0} 2023-05-15 14:06:30.864 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:06:30.864 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:30.864 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:30.864 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:30.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:06:30.0, currentDate=Mon May 15 14:06:30 CST 2023 2023-05-15 14:06:30.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:06:30.0} 2023-05-15 14:06:30.880 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:06:30.881 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:06:30.881 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:06:31.220 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131914, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:31.221 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131914, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:06:31.217+08:00, endTime=null} 2023-05-15 14:06:31.231 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131915, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:31.232 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131915, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:06:31.228+08:00, endTime=null} 2023-05-15 14:06:31.242 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131914, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116310, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:31.217+08:00, sub_task_instance_id=131914}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:06:31.242 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131913, operation:EXECUTE, state: RUNNING 2023-05-15 14:06:31.244 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131913, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:06:31.240+08:00, endTime=null} 2023-05-15 14:06:31.252 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,09d755d08021] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131915, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116311, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:31.228+08:00, sub_task_instance_id=131915}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:06:31.252 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,09d755d08021] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:06:31.268 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131913, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116310, task_operation=execute, latest_execution_start_time=2023-05-15T14:06:31.240+08:00, sub_task_instance_id=131913}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:06:31.271 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:06:31.271 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:06:31.272 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:06:31.272 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:06:31.272 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:06:31.272 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:06:31.283 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.284 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.284 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.284 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.284 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.285 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.286 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.286 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.286 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.286 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.286 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.287 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.288 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-1,fb27d19334114ed7,dcf438736ef0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:06:31.289 INFO 11 --- [pool-3245112-thread-4,fb27d19334114ed7,a968c5bd2524] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:06:31.290 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:06:31.290 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.290 INFO 11 --- [pool-3245112-thread-3,fb27d19334114ed7,1f92436cbd4d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:06:31.290 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.290 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.291 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.291 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.291 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.291 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.292 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.292 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:06:31.292 INFO 11 --- [pool-3245112-thread-2,fb27d19334114ed7,bb7a85b85b14] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:06:31.297 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.297 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.297 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:06:31.298 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.298 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.299 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.303 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.303 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.303 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.303 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:06:31.304 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:06:31.305 INFO 11 --- [pool-3245112-thread-6,fb27d19334114ed7,2ff739d7cfea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:06:31.307 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.308 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:06:31.308 INFO 11 --- [pool-3245112-thread-5,fb27d19334114ed7,7d03c3172cc0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:06:31.309 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,09d755d08021] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:06:31.314 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,09d755d08021] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131915, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:31.314 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.314 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.315 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.315 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:06:31.315 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.316 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:06:31.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.328 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.329 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.329 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.341 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.342 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.342 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.345 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.345 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.346 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.350 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.350 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.351 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.370 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.370 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.370 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:06:31.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.376 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.381 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.381 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.381 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:06:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.393 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.393 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:06:31.396 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:06:31.409 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.409 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.410 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:06:31.418 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:06:31.429 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.430 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.430 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.431 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.431 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.437 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.437 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.437 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.437 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.438 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.442 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.443 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.443 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.446 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.446 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.447 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.459 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.460 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.460 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.460 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.461 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.461 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.462 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.462 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.462 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.462 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.462 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.463 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.463 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.468 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.468 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.468 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.483 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.484 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.484 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.484 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.485 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.487 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.487 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.487 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.501 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.502 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.502 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.502 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.503 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.504 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.504 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.504 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.506 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.507 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.508 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.508 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.509 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.514 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.514 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.515 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.530 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.530 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.531 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.531 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.531 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.532 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.532 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.533 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.533 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.534 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.535 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.535 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.536 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.540 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.540 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.541 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.541 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:06:31.546 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.546 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.546 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.546 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.547 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.548 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.548 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.548 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.550 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:06:31.563 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.563 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.563 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.563 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.564 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.564 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.565 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.565 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:06:31.574 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.575 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.576 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:06:31.577 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:06:31.579 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:06:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.592 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:31.593 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.593 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.593 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.593 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:31.595 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.595 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.595 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.597 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.598 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:06:31.599 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:31.599 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:06:31.603 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.606 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:06:31.606 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.606 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.606 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:06:31.610 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.613 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:06:31.613 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:06:31.613 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:06:31.615 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.617 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:06:31.617 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.617 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.617 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:06:31.618 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2898339ebaa6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131914, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:31.630 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.632 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:06:31.633 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:31.633 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:31.633 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:06:31.642 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:06:31.644 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:06:31.644 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:06:31.645 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:06:31.650 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c4ac025c6e72] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131913, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:06:31.868 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116310, status:SUCCESS 2023-05-15 14:06:31.869 INFO 11 --- [pool-ocp-schedules-19,,32dc8d1f4fad] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:31.869 INFO 11 --- [pool-ocp-schedules-19,,32dc8d1f4fad] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:31.870 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116310, status:SUCCESS 2023-05-15 14:06:31.882 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116311, status:SUCCESS 2023-05-15 14:06:31.884 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116311, status:SUCCESS 2023-05-15 14:06:32.017 INFO 11 --- [pool-ocp-schedules-17,,f5e217e459a5] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:06:32.019 INFO 11 --- [pool-ocp-schedules-17,,f5e217e459a5] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:06:32.040 INFO 11 --- [pool-ocp-schedules-12,,2f81a0ff09b5] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:06:32.042 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.042 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.042 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.042 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.043 INFO 11 --- [pool-ocp-schedules-12,,2f81a0ff09b5] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:06:32.043 INFO 11 --- [pool-ocp-schedules-7,,6d9151cc26dd] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:06:32.043 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.044 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.044 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.045 INFO 11 --- [ForkJoinPool-66039-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:06:32.045 INFO 11 --- [pool-ocp-schedules-7,,6d9151cc26dd] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:06:32.048 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.048 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.048 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.049 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.054 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.054 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.055 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.059 INFO 11 --- [ForkJoinPool-66039-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:06:32.076 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.076 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.076 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.077 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.089 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.089 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.090 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:06:32.093 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.094 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.094 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.095 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.099 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.099 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.100 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:06:32.125 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:32.126 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.126 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.126 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.126 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.127 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.127 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.128 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:06:32.129 WARN 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:06:32.129 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.129 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.130 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:32.131 INFO 11 --- [ForkJoinPool-66041-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:06:32.131 INFO 11 --- [ForkJoinPool-66041-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:06:32.131 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:06:32.149 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.149 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:06:32.157 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:32.158 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.158 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.158 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.159 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.163 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.163 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.164 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:06:32.166 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.167 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.167 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.168 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:06:32.170 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:06:32.179 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:32.179 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.179 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.179 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.180 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.181 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:06:32.181 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.181 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.185 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.185 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:06:32.192 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:06:32.192 INFO 11 --- [ForkJoinPool-66041-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:06:32.201 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.201 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:06:32.213 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.213 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:06:32.228 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.229 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:06:32.244 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.245 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:06:32.262 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:06:32.277 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:06:32.278 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.278 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.278 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.279 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.284 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:06:32.284 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:06:32.284 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:06:32.306 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:06:32.306 INFO 11 --- [ForkJoinPool-66041-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:06:32.800 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:32.802 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:34.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:34.830750+08:00, lastProcessTime=2023-05-15T14:06:24.833711+08:00 2023-05-15 14:06:34.869 INFO 11 --- [alarm-detect-11,,cfd2f16b7d72] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130774 ~ 1684130784 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130780=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:34.872 INFO 11 --- [alarm-detect-11,,cfd2f16b7d72] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:06:34.876 INFO 11 --- [alarm-detect-15,,19daac8af4ca] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130724 ~ 1684130784 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389157.938, values={1684130760=389157.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:06:34.876 INFO 11 --- [alarm-detect-15,,19daac8af4ca] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130724 ~ 1684130784 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=993955.107, values={1684130760=993955.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:34.878 INFO 11 --- [alarm-detect-15,,19daac8af4ca] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:06:34.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:34.830750+08:00 2023-05-15 14:06:35.580 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,3e0ea53f06ff4094,bb7eacc7b85f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=3e0ea53f06ff4094, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:06:35.581 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,3e0ea53f06ff4094,bb7eacc7b85f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=3e0ea53f06ff4094, duration=2 ms] 2023-05-15 14:06:35.802 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:35.803 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:36.006 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:36.009 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:36.011 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:36.020 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:36.994 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:37.929 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sessions?page=1&size=10&activeOnly=false, client=192.168.80.229, traceId=9a2ff410abbc4443, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.listTenantSessions(Pageable,Long,Long,String,String,String,Boolean), args=Page request [number: 0, size 10, sort: id: DESC],2,6,false,] 2023-05-15 14:06:37.958 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:06:37.959 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:06:37.973 ERROR 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573887) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573887) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:37.975 ERROR 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] com.alibaba.druid.pool.DruidDataSource : {dataSource-98} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573887) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573887) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:37.975 INFO 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] com.alibaba.druid.pool.DruidDataSource : {dataSource-98,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:06:37.975 INFO 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] com.alibaba.druid.pool.DruidDataSource : {dataSource-98} closing ... 2023-05-15 14:06:37.976 INFO 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] com.alibaba.druid.pool.DruidDataSource : {dataSource-98} closed 2023-05-15 14:06:37.979 ERROR 11 --- [obsdk-init-thread-15,9a2ff410abbc4443,c8a80ba11353] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573887) Get Location Cache Fail 2023-05-15 14:06:37.979 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573887) Get Location Cache Fail 2023-05-15 14:06:37.980 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573887) Get Location Cache Fail 2023-05-15 14:06:37.980 WARN 11 --- [http-nio-0.0.0.0-8080-exec-66,9a2ff410abbc4443,5f35941d59de] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/sessions client=192.168.80.229, traceId=9a2ff410abbc4443, duration=55 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:06:37.988 ERROR 11 --- [Druid-ConnectionPool-Create-1478335884,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573890) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573890) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:06:38.267 INFO 11 --- [pool-overview-container-5,,de05a6b73b56] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=3 2023-05-15 14:06:38.267 INFO 11 --- [pool-overview-container-5,,de05a6b73b56] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:38.308 INFO 11 --- [pool-ocp-schedules-3,,33c2af0296c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:38.514 INFO 11 --- [pool-scanner--15973231-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:38.803 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:38.804 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:39.304 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,81ca2df62a984836,a778ce145750] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=81ca2df62a984836, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:39.310 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,81ca2df62a984836,a778ce145750] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=81ca2df62a984836, duration=6 ms] 2023-05-15 14:06:39.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,555a4cd9e1324725,6aa98697c9b5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=555a4cd9e1324725, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:39.704 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,555a4cd9e1324725,6aa98697c9b5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=555a4cd9e1324725, duration=6 ms] 2023-05-15 14:06:40.317 INFO 11 --- [pool-ocp-schedules-18,,ef4d08a50fa0] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=3 2023-05-15 14:06:41.805 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:41.806 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:42.395 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:44.806 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:44.807 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:44.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:44.831520+08:00, lastProcessTime=2023-05-15T14:06:34.830750+08:00 2023-05-15 14:06:44.860 INFO 11 --- [alarm-detect-6,,19ca0e29f46d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130784 ~ 1684130794 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130790=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:44.891 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:44.831520+08:00 2023-05-15 14:06:45.484 INFO 11 --- [pool-ocp-schedules-18,,ff941f950eca] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:06:46.009 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:46.011 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:46.013 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:46.022 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:46.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,69b12e5075d74c7d,1dab78ac2bd9] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=69b12e5075d74c7d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:46.453 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,69b12e5075d74c7d,1dab78ac2bd9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=69b12e5075d74c7d, duration=6 ms] 2023-05-15 14:06:46.934 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931} 2023-05-15 14:06:46.971 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.971 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.978 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.979 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.980 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.980 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.981 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.982 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.989 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.991 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.992 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:02:46.866, end=2023-05-15T14:04:46.931},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:06:46.992 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:06:46.992 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:06:46.997 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:47.807 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:47.808 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:48.265 INFO 11 --- [pool-overview-container-2,,3e70b74d19ed] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:48.331 INFO 11 --- [pool-ocp-schedules-13,,c60ab2e58782] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:48.513 INFO 11 --- [pool-scanner--15973331-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:50.809 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:50.810 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:51.819 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:51.822 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:51.824 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:51.874 INFO 11 --- [pool-ocp-schedules-9,,547972fb5b6e] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:51.874 INFO 11 --- [pool-ocp-schedules-9,,547972fb5b6e] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:06:52.791 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/parameters?null, client=192.168.80.229, traceId=33aa1361577645ad, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantParameterController.listTenantParameters(Long,Long,Boolean,Boolean), args=2,6,] 2023-05-15 14:06:52.810 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:06:52.810 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:06:52.819 ERROR 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573893) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573893) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:52.820 ERROR 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] com.alibaba.druid.pool.DruidDataSource : {dataSource-99} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573893) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573893) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:52.820 INFO 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] com.alibaba.druid.pool.DruidDataSource : {dataSource-99,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:06:52.820 INFO 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] com.alibaba.druid.pool.DruidDataSource : {dataSource-99} closing ... 2023-05-15 14:06:52.820 INFO 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] com.alibaba.druid.pool.DruidDataSource : {dataSource-99} closed 2023-05-15 14:06:52.820 ERROR 11 --- [obsdk-init-thread-16,33aa1361577645ad,d861dd6563a8] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573893) Get Location Cache Fail 2023-05-15 14:06:52.823 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573893) Get Location Cache Fail 2023-05-15 14:06:52.823 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573893) Get Location Cache Fail 2023-05-15 14:06:52.824 WARN 11 --- [http-nio-0.0.0.0-8080-exec-56,33aa1361577645ad,cd4aa0d7f6f2] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/parameters client=192.168.80.229, traceId=33aa1361577645ad, duration=34 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:06:52.833 ERROR 11 --- [Druid-ConnectionPool-Create-593750483,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573896) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573896) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:06:53.420 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sessions?page=1&size=10&activeOnly=false, client=192.168.80.229, traceId=44820c45fc514219, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.listTenantSessions(Pageable,Long,Long,String,String,String,Boolean), args=Page request [number: 0, size 10, sort: id: DESC],2,6,false,] 2023-05-15 14:06:53.444 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:06:53.444 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:06:53.455 ERROR 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573899) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573899) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:53.456 ERROR 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] com.alibaba.druid.pool.DruidDataSource : {dataSource-100} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573899) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573899) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:06:53.456 INFO 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] com.alibaba.druid.pool.DruidDataSource : {dataSource-100,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:06:53.457 INFO 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] com.alibaba.druid.pool.DruidDataSource : {dataSource-100} closing ... 2023-05-15 14:06:53.457 INFO 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] com.alibaba.druid.pool.DruidDataSource : {dataSource-100} closed 2023-05-15 14:06:53.460 ERROR 11 --- [obsdk-init-thread-1,44820c45fc514219,23d420c69404] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573899) Get Location Cache Fail 2023-05-15 14:06:53.461 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573899) Get Location Cache Fail 2023-05-15 14:06:53.461 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573899) Get Location Cache Fail 2023-05-15 14:06:53.461 WARN 11 --- [http-nio-0.0.0.0-8080-exec-64,44820c45fc514219,4d9ed950ffc2] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/sessions client=192.168.80.229, traceId=44820c45fc514219, duration=45 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:06:53.477 ERROR 11 --- [Druid-ConnectionPool-Create-2065252838,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573902) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573902) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:06:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,95b6f9f4fa98] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:06:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,95b6f9f4fa98] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:06:53.594 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,d50637c9a47b44a2,651a18485b34] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=d50637c9a47b44a2, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:53.600 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,d50637c9a47b44a2,651a18485b34] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=d50637c9a47b44a2, duration=6 ms] 2023-05-15 14:06:53.810 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:53.811 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:06:54.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:06:54.828925+08:00, lastProcessTime=2023-05-15T14:06:44.831520+08:00 2023-05-15 14:06:54.884 INFO 11 --- [alarm-detect-6,,627cfc904180] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130794 ~ 1684130804 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130800=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:06:54.923 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:06:54.828925+08:00 2023-05-15 14:06:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,9aed8ff7bc9c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:06:55.102 INFO 11 --- [pool-prometheus-second-schedule-2,,6014261737ef] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:06:55.325 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,7a51c7363bec43c8,b757bae099ea] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=7a51c7363bec43c8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:06:55.331 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,7a51c7363bec43c8,b757bae099ea] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=7a51c7363bec43c8, duration=6 ms] 2023-05-15 14:06:56.011 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:56.014 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:56.016 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:56.024 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:56.811 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:56.812 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:06:56.999 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:57.398 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:06:58.265 INFO 11 --- [pool-overview-container-7,,3fd558528c1c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:06:58.353 INFO 11 --- [pool-ocp-schedules-10,,bef8590b9c42] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:06:58.514 INFO 11 --- [pool-scanner--15973425-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:06:59.812 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:06:59.813 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:00.000 INFO 11 --- [pool-ocp-schedules-2,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:07:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:07:00.000 INFO 11 --- [pool-ocp-schedules-2,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130820 2023-05-15 14:07:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:07:00.050 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:07:00.765 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:07:00.0, currentDate=Mon May 15 14:07:00 CST 2023 2023-05-15 14:07:00.765 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:07:00.0} 2023-05-15 14:07:00.768 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:07:00.768 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:00.768 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:00.776 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:07:00.0, currentDate=Mon May 15 14:07:00 CST 2023 2023-05-15 14:07:00.776 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:07:00.0} 2023-05-15 14:07:00.787 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:07:00.787 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:00.787 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:00.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:07:00.0, currentDate=Mon May 15 14:07:00 CST 2023 2023-05-15 14:07:00.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:07:00.0} 2023-05-15 14:07:00.805 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:07:00.805 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:00.805 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:00.810 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:31.0, nextRun=2023-05-15 14:07:00.0, currentDate=Mon May 15 14:07:00 CST 2023 2023-05-15 14:07:00.810 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:06:31.0, nextRunTime=2023-05-15 14:07:00.0} 2023-05-15 14:07:00.813 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:07:00.813 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:00.813 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:00.819 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:07:00.0, currentDate=Mon May 15 14:07:00 CST 2023 2023-05-15 14:07:00.819 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:07:00.0} 2023-05-15 14:07:00.822 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:07:00.822 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:00.822 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,1fd269889b23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130880, interval=60 2023-05-15 14:07:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,2b9a1a02ad73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130880, interval=60 2023-05-15 14:07:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,1fd269889b23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:07:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,1fd269889b23] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130821, count=8, elapsed=1682446690703 2023-05-15 14:07:01.524 INFO 11 --- [pool-prometheus-minute-schedule-1,,2b9a1a02ad73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:07:01.524 INFO 11 --- [pool-prometheus-minute-schedule-1,,2b9a1a02ad73] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130821, count=6, elapsed=1682446690703 2023-05-15 14:07:01.672 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131920, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:01.673 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131920, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:07:01.670+08:00, endTime=null} 2023-05-15 14:07:01.683 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131918, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:01.684 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131918, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:07:01.680+08:00, endTime=null} 2023-05-15 14:07:01.690 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,55541a990f13] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131920, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116316, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:01.670+08:00, sub_task_instance_id=131920}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:01.695 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131919, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:01.697 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131919, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:07:01.693+08:00, endTime=null} 2023-05-15 14:07:01.703 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,04d4935acf3e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131918, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116314, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:01.680+08:00, sub_task_instance_id=131918}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:01.706 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,55541a990f13] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131920, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:01.711 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131916, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:01.713 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131916, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:07:01.707+08:00, endTime=null} 2023-05-15 14:07:01.717 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c0cb003f9376] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131919, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116315, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:01.692+08:00, sub_task_instance_id=131919}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:01.717 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c0cb003f9376] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:07:01.726 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131917, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:01.728 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131917, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:07:01.723+08:00, endTime=null} 2023-05-15 14:07:01.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,04d4935acf3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:07:01.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,04d4935acf3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:07:01.732 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,04d4935acf3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:07:01.736 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6eaa65b0f6d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131916, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116312, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:01.707+08:00, sub_task_instance_id=131916}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:01.737 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,04d4935acf3e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131918, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:01.742 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:07:01.742 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:07:01.742 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:07:01.743 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:07:01.743 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:07:01.745 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:07:01.746 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:07:01.746 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:07:01.746 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:07:01.746 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:07:01.748 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:07:01.753 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:07:01.756 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:07:01.756 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,6eaa65b0f6d0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131916, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:01.757 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:07:01.758 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:07:01.759 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,0da4df5d03d4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131917, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116313, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:01.723+08:00, sub_task_instance_id=131917}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:01.761 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:07:01.764 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.766 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.766 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.766 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.766 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.768 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.769 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.769 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.769 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.769 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.770 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.771 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.771 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.772 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.773 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.773 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:07:01.773 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.774 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.774 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.774 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.774 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.775 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:07:01.776 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.776 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.776 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.776 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.776 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.777 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.777 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:07:01.780 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:07:01.780 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:01.780 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.780 INFO 11 --- [pool-3245275-thread-4,fb27d19334114ed7,645112fd5f5b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:07:01.780 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:07:01.781 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:07:01.781 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:07:01.781 INFO 11 --- [pool-3245275-thread-2,fb27d19334114ed7,570689328116] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:07:01.781 INFO 11 --- [pool-3245275-thread-3,fb27d19334114ed7,f553c2cac2c8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:07:01.784 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:07:01.784 INFO 11 --- [pool-3245275-thread-1,fb27d19334114ed7,de8b966f7b5a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:07:01.785 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.788 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:07:01.789 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,0da4df5d03d4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:07:01.790 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,0da4df5d03d4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:07:01.792 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.792 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.793 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:01.794 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:07:01.795 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.795 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.796 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,0da4df5d03d4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131917, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:01.796 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:07:01.797 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:07:01.797 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:07:01.797 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:01.797 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:01.798 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:07:01.811 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:07:01.812 INFO 11 --- [pool-3245275-thread-6,fb27d19334114ed7,09a0c13a1a77] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:07:01.815 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:07:01.815 INFO 11 --- [pool-3245275-thread-5,fb27d19334114ed7,f579b3c5e5bc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:07:01.816 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c0cb003f9376] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:07:01.822 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c0cb003f9376] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131919, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:01.825 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:07:02.469 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116312, status:SUCCESS 2023-05-15 14:07:02.470 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116312, status:SUCCESS 2023-05-15 14:07:02.479 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116313, status:SUCCESS 2023-05-15 14:07:02.480 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116313, status:SUCCESS 2023-05-15 14:07:02.490 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116314, status:SUCCESS 2023-05-15 14:07:02.491 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116314, status:SUCCESS 2023-05-15 14:07:02.500 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116315, status:SUCCESS 2023-05-15 14:07:02.501 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116315, status:SUCCESS 2023-05-15 14:07:02.511 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116316, status:SUCCESS 2023-05-15 14:07:02.512 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116316, status:SUCCESS 2023-05-15 14:07:02.813 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:02.814 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:04.458 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5fd2c0837ea455e,edd70f4dbb33] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=d5fd2c0837ea455e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:04.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d5fd2c0837ea455e,edd70f4dbb33] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=d5fd2c0837ea455e, duration=5 ms] 2023-05-15 14:07:04.714 INFO 11 --- [pool-ocp-schedules-7,,96674431336a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:07:04.721 INFO 11 --- [pool-ocp-schedules-7,,96674431336a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:07:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:04.831173+08:00, lastProcessTime=2023-05-15T14:06:54.828925+08:00 2023-05-15 14:07:04.866 INFO 11 --- [alarm-detect-13,,1a308e3a6dc7] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130804 ~ 1684130814 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130810=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:04.891 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:04.831173+08:00 2023-05-15 14:07:05.814 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:05.816 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:05.944 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:05.0, nextRun=2023-05-15 14:07:05.0, currentDate=Mon May 15 14:07:05 CST 2023 2023-05-15 14:07:05.944 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:06:05.0, nextRunTime=2023-05-15 14:07:05.0} 2023-05-15 14:07:05.949 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:07:05.949 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:05.949 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:06.014 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:06.017 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:06.019 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:06.025 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:06.812 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131921, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:06.814 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131921, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:07:06.809+08:00, endTime=null} 2023-05-15 14:07:06.833 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22e210528d03] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131921, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116317, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:06.809+08:00, sub_task_instance_id=131921}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:06.857 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,4287f14d2875] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:07:06.858 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,a8aeab092f71] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:07:06.859 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,8dbec5cd6e1d] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:07:06.896 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,8dbec5cd6e1d] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:07:06.899 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,4287f14d2875] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:07:06.899 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,a8aeab092f71] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:07:06.909 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,22e210528d03] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131921, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:07.003 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:07.613 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116317, status:SUCCESS 2023-05-15 14:07:07.614 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116317, status:SUCCESS 2023-05-15 14:07:08.265 INFO 11 --- [pool-overview-container-1,,7cb3904d06de] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:08.373 INFO 11 --- [pool-ocp-schedules-17,,8f8f15df77a5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:08.512 INFO 11 --- [pool-scanner--15973518-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:07:08.816 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:08.819 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:09.613 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,579c8a77d67941d3,666e9382ad3b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=579c8a77d67941d3, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:07:09.615 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,579c8a77d67941d3,666e9382ad3b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=579c8a77d67941d3, duration=2 ms] 2023-05-15 14:07:09.947 INFO 11 --- [pool-ocp-schedules-10,,113878646ebf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:07:09.948 INFO 11 --- [pool-ocp-schedules-10,,113878646ebf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:07:11.820 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:11.821 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:11.879 INFO 11 --- [pool-ocp-schedules-10,,3be8f2908e27] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:11.879 INFO 11 --- [pool-ocp-schedules-10,,3be8f2908e27] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:12.402 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:13.202 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8e9081d629db43cf,007288592e4f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8e9081d629db43cf, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:13.208 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8e9081d629db43cf,007288592e4f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8e9081d629db43cf, duration=6 ms] 2023-05-15 14:07:14.453 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130760, nodeCount=244, affectRows=244 2023-05-15 14:07:14.456 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130760, nodeCount=244, validCount=244 2023-05-15 14:07:14.473 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130760, nodeCount=1000, affectRows=1000 2023-05-15 14:07:14.478 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130760, nodeCount=724, affectRows=724 2023-05-15 14:07:14.479 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130760, nodeCount=722, affectRows=722 2023-05-15 14:07:14.482 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130760, nodeCount=1000, validCount=1000 2023-05-15 14:07:14.485 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130760, nodeCount=724, validCount=724 2023-05-15 14:07:14.485 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130760, nodeCount=722, validCount=722 2023-05-15 14:07:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:14.821 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:14.822 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:14.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:14.828170+08:00, lastProcessTime=2023-05-15T14:07:04.831173+08:00 2023-05-15 14:07:14.875 INFO 11 --- [alarm-detect-11,,32ade9dd3d0d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130814 ~ 1684130824 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.949, values={1684130820=245.949}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:14.907 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:14.828170+08:00 2023-05-15 14:07:16.017 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:16.020 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:16.022 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:16.028 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:16.984 INFO 11 --- [pool-ocp-schedules-14,,2b6cbcf33ed9] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:07:16.984 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:07:16.984 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:07:16.984 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:07:16.984 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:07:16.985 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:07:16.990 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:07:16.991 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:07:16.992 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:07:16.992 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:07:16.992 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:07:17.005 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:17.018 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:07:17.018 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:07:17.020 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:07:17.021 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:07:17.020586541+08:00 2023-05-15 14:07:17.021 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:07:17.020611071+08:00 2023-05-15 14:07:17.021 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:07:17.021+08:00 2023-05-15 14:07:17.021 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:07:17.021+08:00 2023-05-15 14:07:17.022 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:07:17.021922419+08:00 2023-05-15 14:07:17.022 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:07:17.022+08:00 2023-05-15 14:07:17.023 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:07:17.023 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:07:17.024 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:07:17.021+08:00 and status AVAILABLE 2023-05-15 14:07:17.024 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:07:17.021+08:00 and status AVAILABLE 2023-05-15 14:07:17.025 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:07:17.024718270+08:00 2023-05-15 14:07:17.025 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:07:17.025+08:00 2023-05-15 14:07:17.025 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:07:17.025341865+08:00 2023-05-15 14:07:17.025 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:07:17.025+08:00 2023-05-15 14:07:17.026 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:07:17.022+08:00 and status AVAILABLE 2023-05-15 14:07:17.028 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:07:17.025+08:00 and status AVAILABLE 2023-05-15 14:07:17.029 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:07:17.025+08:00 and status AVAILABLE 2023-05-15 14:07:17.030 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:07:17.030 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:07:17.033 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:07:17.055 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130837 0 0 , error= 2023-05-15 14:07:17.055 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130837 0 0 , error= 2023-05-15 14:07:17.056 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:07:17.056+08:00 2023-05-15 14:07:17.056 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:07:17.056+08:00 2023-05-15 14:07:17.057 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:07:17.057 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:07:17.074 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130837 0 0 , error= 2023-05-15 14:07:17.075 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:07:17.075+08:00 2023-05-15 14:07:17.075 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:07:17.056+08:00 and status AVAILABLE 2023-05-15 14:07:17.087 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130837 0 0 , error= 2023-05-15 14:07:17.096 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130837 0 0 , error= 2023-05-15 14:07:17.096 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:07:17.056+08:00 and status AVAILABLE 2023-05-15 14:07:17.097 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:07:17.096+08:00 2023-05-15 14:07:17.097 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:07:17.096+08:00 2023-05-15 14:07:17.101 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:07:17.101 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:07:17.096+08:00 and status AVAILABLE 2023-05-15 14:07:17.101 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:07:17.075+08:00 and status AVAILABLE 2023-05-15 14:07:17.101 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:07:17.102 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:07:17.096+08:00 and status AVAILABLE 2023-05-15 14:07:17.103 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:07:17.103 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:07:17.104 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:07:17.110 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:07:17.111 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:07:17.113 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:07:17.114 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:07:17.115 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:07:17.135 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:07:17.137 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:07:17.138 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:07:17.138 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:07:17.142 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:07:17.180 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:07:17.187 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:07:17.190 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:07:17.192 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:07:17.192 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:07:17.202 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:07:17.202 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:07:17.208 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:07:17.210 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:07:17.211 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:07:17.218 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:07:17.221 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:07:17.223 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:07:17.227 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:07:17.242 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:07:17.252 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:07:17.269 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:07:17.270 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:07:17.271 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:07:17.282 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:07:17.285 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:07:17.285 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:07:17.290 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:07:17.297 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:07:17.306 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:07:17.308 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:07:17.316 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:07:17.320 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:07:17.337 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:07:17.339 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:07:17.351 INFO 11 --- [pool-Check host status1,,d1d68df69a58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:07:17.361 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:07:17.363 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:07:17.371 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:07:17.376 INFO 11 --- [pool-Check host status14,,a262acf2cb65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:07:17.377 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:07:17.385 INFO 11 --- [pool-Check host status30,,d2c17638bb40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:07:17.391 INFO 11 --- [pool-Check host status8,,5a5964309670] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:07:17.448 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:07:17.464 INFO 11 --- [pool-Check host status13,,de414c828057] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:07:17.483 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:07:17.484 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:07:17.484 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:07:17.484 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:07:17.484 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:07:17.489 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:07:17.490 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:07:17.491 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:07:17.491 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:07:17.493 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:07:17.518 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:07:17.520 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:07:17.520 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:07:17.521 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:07:17.525 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:07:17.539 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:07:17.539 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:07:17.539 INFO 11 --- [pool-Check host status17,,24ec0a7dc464] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:07:17.540 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:07:17.540 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:07:17.540 INFO 11 --- [pool-Check host status3,,b01cb0319cb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:07:17.540 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:07:17.540 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:07:17.541 INFO 11 --- [pool-Check host status28,,39418887c135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:07:17.542 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:07:17.542 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:07:17.542 INFO 11 --- [pool-Check host status2,,6789d195f04c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:07:17.545 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:07:17.545 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:07:17.545 INFO 11 --- [pool-Check host status19,,bd54a98eb1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:07:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:07:17.623 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.624 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:17.626 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:17.626 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.626 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:07:17.648 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:17.649 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:17.650 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:17.650 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.651 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:17.656 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:17.656 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.657 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:07:17.659 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:07:17.659 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:07:17.659 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:07:17.659 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:07:17.665 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:07:17.665 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.665 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:07:17.665 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.666 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:07:17.666 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:17.666 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:07:17.667 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:07:17.667 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:07:17.689 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:04:48, start=2023-05-15T14:00:47, end=2023-05-15T14:03:17, updates=96, granule=120000ms, delay=241000ms 2023-05-15 14:07:17.691 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:04:48, target=2023-05-15T14:03:17, granule=120000ms, delay=91000ms) 2023-05-15 14:07:17.692 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:05:02, start=2023-05-15T14:00:47, end=2023-05-15T14:03:17, updates=100, granule=120000ms, delay=255000ms 2023-05-15 14:07:17.694 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:04:48, start=2023-05-15T14:00:47, end=2023-05-15T14:03:17, updates=127, granule=120000ms, delay=241000ms 2023-05-15 14:07:17.694 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:05:02, target=2023-05-15T14:03:17, granule=120000ms, delay=105000ms) 2023-05-15 14:07:17.694 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:03:17, target=2023-05-15T13:53:17, granule=600000ms, delay=600000ms) 2023-05-15 14:07:17.695 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:04:48, target=2023-05-15T14:03:17, granule=120000ms, delay=91000ms) 2023-05-15 14:07:17.779 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=1->2, c=0) - source=2023-05-15T14:03:17, start=2023-05-15T13:51:17, end=2023-05-15T14:03:17, updates=128, granule=600000ms, delay=720000ms 2023-05-15 14:07:17.781 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=1) - not enough cached interval (source=2023-05-15T14:03:17, target=2023-05-15T14:03:17, granule=600000ms, delay=0ms) 2023-05-15 14:07:17.787 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=1->2, c=0) - source=2023-05-15T14:03:17, start=2023-05-15T13:51:17, end=2023-05-15T14:03:17, updates=179, granule=600000ms, delay=720000ms 2023-05-15 14:07:17.788 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=1) - not enough cached interval (source=2023-05-15T14:03:17, target=2023-05-15T14:03:17, granule=600000ms, delay=0ms) 2023-05-15 14:07:17.788 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=201ms, period=120000ms 2023-05-15 14:07:17.822 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:17.823 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:18.265 INFO 11 --- [pool-overview-container-5,,76f21b2c42cb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:18.395 INFO 11 --- [pool-ocp-schedules-2,,a756f949340c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:18.512 INFO 11 --- [pool-scanner--15973618-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:20.441 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:20.0, nextRun=2023-05-15 14:07:20.0, currentDate=Mon May 15 14:07:20 CST 2023 2023-05-15 14:07:20.441 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:06:20.0, nextRunTime=2023-05-15 14:07:20.0} 2023-05-15 14:07:20.449 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:07:20.449 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:20.449 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:20.823 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:20.825 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:21.022 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131922, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:21.023 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131922, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:07:21.019+08:00, endTime=null} 2023-05-15 14:07:21.046 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c61035587da8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131922, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116318, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:21.019+08:00, sub_task_instance_id=131922}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:21.048 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c61035587da8] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:07:21.049 INFO 11 --- [ForkJoinPool-66044-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:07:21.049 INFO 11 --- [ForkJoinPool-66044-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:07:21.055 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c61035587da8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131922, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:21.819 INFO 11 --- [pool-scanner--15973619-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:21.822 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:21.824 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:21.826 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:21.842 INFO 11 --- [pool-scanner--15973620-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:21.864 INFO 11 --- [pool-scanner--15973621-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:21.879 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116318, status:SUCCESS 2023-05-15 14:07:21.881 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116318, status:SUCCESS 2023-05-15 14:07:21.885 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:21.890 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:21.934 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:21.938 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:21.958 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:21.963 INFO 11 --- [pool-ocp-schedules-12,,fec780284095] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:23.825 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:23.826 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:24.830719+08:00, lastProcessTime=2023-05-15T14:07:14.828170+08:00 2023-05-15 14:07:24.906 INFO 11 --- [alarm-detect-15,,cd7af7b3dc36] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130824 ~ 1684130834 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130830=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:24.910 INFO 11 --- [alarm-detect-3,,e8d3af8a34a0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130774 ~ 1684130834 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389217.988, values={1684130820=389217.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:07:24.917 INFO 11 --- [alarm-detect-3,,e8d3af8a34a0] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130774 ~ 1684130834 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994015.358, values={1684130820=994015.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:24.920 INFO 11 --- [alarm-detect-3,,e8d3af8a34a0] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:07:24.955 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:24.830719+08:00 2023-05-15 14:07:25.612 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:26.0, nextRun=2023-05-15 14:07:25.0, currentDate=Mon May 15 14:07:25 CST 2023 2023-05-15 14:07:25.612 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:06:26.0, nextRunTime=2023-05-15 14:07:25.0} 2023-05-15 14:07:25.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:07:25.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:25.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:26.020 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:26.022 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:26.024 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:26.029 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:26.112 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131923, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:26.115 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131923, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:07:26.109+08:00, endTime=null} 2023-05-15 14:07:26.137 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131923, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116319, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:26.109+08:00, sub_task_instance_id=131923}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:07:26.179 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:07:26.180 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:07:26.181 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:07:26.181 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:07:26.182 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:07:26.186 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,2d7ff504d539] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131923, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:26.826 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:26.827 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:26.997 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116319, status:SUCCESS 2023-05-15 14:07:26.998 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116319, status:SUCCESS 2023-05-15 14:07:27.008 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:27.406 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:27.495 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,268c69e6e0c04571,b3627e5793d3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=268c69e6e0c04571, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:27.501 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,268c69e6e0c04571,b3627e5793d3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=268c69e6e0c04571, duration=7 ms] 2023-05-15 14:07:28.266 INFO 11 --- [pool-overview-container-3,,694c21ec0df4] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:07:28.266 INFO 11 --- [pool-overview-container-3,,694c21ec0df4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:28.416 INFO 11 --- [pool-ocp-schedules-19,,99dc9de2c994] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:28.512 INFO 11 --- [pool-scanner--15973719-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:29.828 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:29.829 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:30.348 INFO 11 --- [pool-ocp-schedules-11,,fe351b89c5b3] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:07:30.773 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:31.0, nextRun=2023-05-15 14:07:30.0, currentDate=Mon May 15 14:07:30 CST 2023 2023-05-15 14:07:30.773 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:06:31.0, nextRunTime=2023-05-15 14:07:30.0} 2023-05-15 14:07:30.781 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:07:30.781 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:30.781 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:30.781 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:30.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:01.0, nextRun=2023-05-15 14:07:30.0, currentDate=Mon May 15 14:07:30 CST 2023 2023-05-15 14:07:30.792 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:07:01.0, nextRunTime=2023-05-15 14:07:30.0} 2023-05-15 14:07:30.794 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:07:30.794 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:07:30.794 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:07:31.195 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131926, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:31.197 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131926, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:07:31.192+08:00, endTime=null} 2023-05-15 14:07:31.212 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131924, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:31.214 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131924, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:07:31.208+08:00, endTime=null} 2023-05-15 14:07:31.216 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8a27c43dedbd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131926, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116321, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:31.192+08:00, sub_task_instance_id=131926}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:07:31.216 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8a27c43dedbd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:07:31.226 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131925, operation:EXECUTE, state: RUNNING 2023-05-15 14:07:31.228 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131925, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:07:31.224+08:00, endTime=null} 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:07:31.234 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:07:31.235 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131924, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116320, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:31.208+08:00, sub_task_instance_id=131924}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:07:31.245 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.246 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.246 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.246 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.247 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.248 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.248 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.248 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.248 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.248 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:07:31.249 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.250 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131925, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116320, task_operation=execute, latest_execution_start_time=2023-05-15T14:07:31.224+08:00, sub_task_instance_id=131925}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:07:31.250 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.250 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.251 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:07:31.251 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:07:31.251 INFO 11 --- [pool-3245433-thread-4,fb27d19334114ed7,c376fe066b77] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:07:31.251 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:07:31.252 INFO 11 --- [pool-3245433-thread-2,fb27d19334114ed7,51888540e273] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:07:31.252 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-1,fb27d19334114ed7,3ff5e2c9cec3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-3,fb27d19334114ed7,d8f18520d688] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:07:31.253 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.254 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.254 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.254 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.255 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.259 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.259 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.260 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.260 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.260 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:07:31.260 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:07:31.264 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:07:31.264 INFO 11 --- [pool-3245433-thread-6,fb27d19334114ed7,d59c326652a9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:07:31.265 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:07:31.266 INFO 11 --- [pool-3245433-thread-5,fb27d19334114ed7,04721d9535bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:07:31.266 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8a27c43dedbd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:07:31.270 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.271 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.271 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.271 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.271 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8a27c43dedbd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131926, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:31.272 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.272 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.272 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:07:31.280 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.280 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.280 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.292 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.292 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.292 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.307 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.308 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.312 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.312 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.312 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.313 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.313 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.313 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:07:31.318 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.321 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.321 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.321 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.322 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.322 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:07:31.332 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.332 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.332 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:07:31.338 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.338 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.339 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.343 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.343 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.343 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:07:31.350 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.350 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.351 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.353 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.353 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.354 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:07:31.363 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:07:31.364 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.364 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.364 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.377 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:07:31.380 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.381 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.381 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.381 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.381 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.382 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.382 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.382 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.395 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.395 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.396 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.404 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.405 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.406 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.410 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.410 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.411 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.412 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.412 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.413 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.413 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.413 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.414 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.414 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.414 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.424 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.424 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.428 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.428 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.429 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.429 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.431 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.435 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.435 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.436 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.446 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.447 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.447 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.447 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.447 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.448 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.448 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.449 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.459 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.459 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.460 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.460 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.460 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.461 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.461 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.462 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.463 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.464 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.465 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.465 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.466 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.470 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.470 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.471 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.482 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.482 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.483 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.483 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.483 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.484 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.485 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.485 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:07:31.492 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.493 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.493 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.493 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.494 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.498 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.499 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.499 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.499 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.500 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.500 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.500 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.501 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.501 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.501 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.501 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:07:31.502 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:07:31.502 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:07:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:07:31.506 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:07:31.530 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.531 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.532 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.532 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.532 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.537 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.537 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.538 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.543 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:31.544 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.544 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.544 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.545 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:31.550 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.550 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.550 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.551 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.552 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:07:31.553 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.553 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.553 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:07:31.564 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.566 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.566 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:07:31.567 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:31.567 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.568 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:07:31.568 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:07:31.568 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.568 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.568 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:07:31.577 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.579 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.579 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:07:31.579 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:31.579 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:31.580 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:07:31.584 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:07:31.584 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:07:31.584 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:07:31.591 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:07:31.591 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,21bf75022e08] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131925, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:31.592 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:07:31.592 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:07:31.593 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:07:31.596 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,751292df7223] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131924, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:07:31.884 INFO 11 --- [pool-ocp-schedules-5,,0077370aa17c] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:31.884 INFO 11 --- [pool-ocp-schedules-5,,0077370aa17c] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:32.023 INFO 11 --- [pool-ocp-schedules-16,,6a3fef241ad5] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:07:32.025 INFO 11 --- [pool-ocp-schedules-16,,6a3fef241ad5] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:07:32.042 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.043 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.043 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.043 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.043 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.043 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.044 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.044 INFO 11 --- [ForkJoinPool-66045-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:07:32.045 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.045 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.045 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.045 INFO 11 --- [pool-ocp-schedules-15,,561e49d3ed3c] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:07:32.046 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.047 INFO 11 --- [pool-ocp-schedules-15,,561e49d3ed3c] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:07:32.048 INFO 11 --- [pool-ocp-schedules-2,,96e2881d3f9d] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:07:32.049 INFO 11 --- [pool-ocp-schedules-2,,96e2881d3f9d] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:07:32.050 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.050 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.050 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.054 INFO 11 --- [ForkJoinPool-66045-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:07:32.072 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.073 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.073 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.073 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.074 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.074 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.075 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:07:32.086 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.086 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.086 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.087 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.091 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.091 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.092 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:07:32.105 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116320, status:SUCCESS 2023-05-15 14:07:32.106 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116320, status:SUCCESS 2023-05-15 14:07:32.116 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:32.116 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.117 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.117 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.117 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.118 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.118 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.118 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:07:32.120 WARN 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:07:32.120 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.120 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.121 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:07:32.121 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116321, status:SUCCESS 2023-05-15 14:07:32.122 INFO 11 --- [ForkJoinPool-66047-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:07:32.122 INFO 11 --- [ForkJoinPool-66047-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:07:32.122 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:07:32.122 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116321, status:SUCCESS 2023-05-15 14:07:32.130 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.130 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:07:32.141 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.153 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:32.153 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.153 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.153 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.154 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.155 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:07:32.155 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.155 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.166 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:07:32.166 INFO 11 --- [ForkJoinPool-66047-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:07:32.166 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:32.167 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.167 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.167 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.168 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.172 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.172 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.173 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:07:32.177 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.177 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.178 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:07:32.180 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:07:32.180 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:07:32.180 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:07:32.180 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:07:32.181 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:07:32.181 INFO 11 --- [ForkJoinPool-66047-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:07:32.181 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:07:32.199 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.199 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:07:32.216 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.216 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:07:32.232 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.233 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:07:32.253 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.253 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:07:32.268 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.268 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:07:32.286 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:07:32.303 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:07:32.305 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.305 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.305 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.306 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.311 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:07:32.311 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:07:32.311 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:07:32.333 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:07:32.333 INFO 11 --- [ForkJoinPool-66047-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:07:32.829 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:32.830 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:33.822 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,b9e790c3919a4435,df42c1d121f3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=b9e790c3919a4435, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:33.827 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,b9e790c3919a4435,df42c1d121f3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=b9e790c3919a4435, duration=5 ms] 2023-05-15 14:07:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:34.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:34.828168+08:00, lastProcessTime=2023-05-15T14:07:24.830719+08:00 2023-05-15 14:07:34.863 INFO 11 --- [alarm-detect-5,,3e18d0db7aeb] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130834 ~ 1684130844 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130840=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:34.869 INFO 11 --- [alarm-detect-10,,829d15dfd058] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130784 ~ 1684130844 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389217.938, values={1684130820=389217.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:07:34.870 INFO 11 --- [alarm-detect-10,,829d15dfd058] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130784 ~ 1684130844 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994015.107, values={1684130820=994015.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:34.977 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:34.828168+08:00 2023-05-15 14:07:35.830 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:35.831 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:36.023 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:36.025 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:36.027 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:36.031 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:36.423 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,ba1d01fe7da94c00,68d055be15c1] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=ba1d01fe7da94c00, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:36.428 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,ba1d01fe7da94c00,68d055be15c1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=ba1d01fe7da94c00, duration=5 ms] 2023-05-15 14:07:37.010 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:38.265 INFO 11 --- [pool-overview-container-4,,6eebf88b952e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:38.441 INFO 11 --- [pool-ocp-schedules-3,,a172913004e3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:38.510 INFO 11 --- [pool-scanner--15973830-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:38.831 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:38.834 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:41.835 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:41.836 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:42.408 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:43.649 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,e7498924a4a94f73,928828591d6a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=e7498924a4a94f73, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:07:43.652 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,e7498924a4a94f73,928828591d6a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e7498924a4a94f73, duration=3 ms] 2023-05-15 14:07:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:44.831926+08:00, lastProcessTime=2023-05-15T14:07:34.828168+08:00 2023-05-15 14:07:44.836 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:44.837 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:44.856 INFO 11 --- [alarm-detect-15,,a512450c4741] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130844 ~ 1684130854 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130850=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:44.901 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:44.831926+08:00 2023-05-15 14:07:45.495 INFO 11 --- [pool-ocp-schedules-3,,7e876bc5992f] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:07:46.025 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:46.028 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:46.030 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:46.033 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:47.018 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:47.102 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,296cbe46c0e84ffa,c8066317f628] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=296cbe46c0e84ffa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:47.108 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,296cbe46c0e84ffa,c8066317f628] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=296cbe46c0e84ffa, duration=7 ms] 2023-05-15 14:07:47.838 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:47.839 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:48.266 INFO 11 --- [pool-overview-container-8,,89b41123f2a4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:48.463 INFO 11 --- [pool-ocp-schedules-18,,1eded084fe37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:48.515 INFO 11 --- [pool-scanner--15973930-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:50.839 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:50.840 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:51.824 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:51.827 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:51.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:51.839 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,03c388e3f1c9432b,ca1893ae7fad] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=03c388e3f1c9432b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:51.845 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,03c388e3f1c9432b,ca1893ae7fad] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=03c388e3f1c9432b, duration=6 ms] 2023-05-15 14:07:51.890 INFO 11 --- [pool-ocp-schedules-19,,ac79a9550203] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:51.890 INFO 11 --- [pool-ocp-schedules-19,,ac79a9550203] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:07:52.047 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,b9a8d00934054699,f1d88c29377f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=b9a8d00934054699, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:07:52.053 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,b9a8d00934054699,f1d88c29377f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=b9a8d00934054699, duration=6 ms] 2023-05-15 14:07:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,ef74305b76f9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:07:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,ef74305b76f9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:07:53.840 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:53.841 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:07:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:07:54.830146+08:00, lastProcessTime=2023-05-15T14:07:44.831926+08:00 2023-05-15 14:07:54.862 INFO 11 --- [alarm-detect-12,,a93bf6c4a532] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130854 ~ 1684130864 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130860=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:07:54.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:07:54.830146+08:00 2023-05-15 14:07:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,08eb44be1a16] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:07:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,1089a4fc60c8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:07:56.028 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:56.031 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:56.033 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:56.036 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:56.842 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:56.843 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:07:57.021 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:57.410 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:07:58.266 INFO 11 --- [pool-overview-container-6,,b41256e05159] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:07:58.485 INFO 11 --- [pool-ocp-schedules-8,,879915a2ba07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:07:58.514 INFO 11 --- [pool-scanner--15974024-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:07:59.843 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:07:59.844 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:08:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:08:00.000 INFO 11 --- [pool-ocp-schedules-14,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130880 2023-05-15 14:08:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:08:00.045 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:08:00.703 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:06:01.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.703 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:06:01.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:08:00.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:00.721 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:01.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.721 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:07:01.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:08:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:00.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:01.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:07:01.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.740 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:08:00.740 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.740 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:00.746 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:01.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.746 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:07:01.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:08:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:00.766 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:31.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.766 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:07:31.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.769 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:08:00.769 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.769 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:00.775 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:01.0, nextRun=2023-05-15 14:08:00.0, currentDate=Mon May 15 14:08:00 CST 2023 2023-05-15 14:08:00.775 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:07:01.0, nextRunTime=2023-05-15 14:08:00.0} 2023-05-15 14:08:00.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:08:00.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:00.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:01.391 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,26d2b28a56874e14,f5f92564d6ba] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=26d2b28a56874e14, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:01.397 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,26d2b28a56874e14,f5f92564d6ba] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=26d2b28a56874e14, duration=6 ms] 2023-05-15 14:08:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,7c7c1d417a22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130940, interval=60 2023-05-15 14:08:01.514 INFO 11 --- [pool-prometheus-minute-schedule-2,,74a003ec2fb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684130940, interval=60 2023-05-15 14:08:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,74a003ec2fb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:08:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,74a003ec2fb3] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130881, count=6, elapsed=1682446750643 2023-05-15 14:08:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,7c7c1d417a22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:08:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,7c7c1d417a22] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130881, count=8, elapsed=1682446750644 2023-05-15 14:08:01.668 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131927, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.669 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131927, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:08:01.665+08:00, endTime=null} 2023-05-15 14:08:01.680 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131930, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.681 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131930, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:08:01.677+08:00, endTime=null} 2023-05-15 14:08:01.690 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,099bae381635] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131927, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116322, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.665+08:00, sub_task_instance_id=131927}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.692 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131931, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.694 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131931, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:08:01.690+08:00, endTime=null} 2023-05-15 14:08:01.702 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,597ea2fc8306] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131930, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116324, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.677+08:00, sub_task_instance_id=131930}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.707 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131928, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.709 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131928, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:08:01.704+08:00, endTime=null} 2023-05-15 14:08:01.711 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,099bae381635] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131927, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.714 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131931, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116325, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.689+08:00, sub_task_instance_id=131931}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.721 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131929, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.724 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131929, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:08:01.719+08:00, endTime=null} 2023-05-15 14:08:01.725 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,597ea2fc8306] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:08:01.726 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,597ea2fc8306] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:08:01.731 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,597ea2fc8306] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131930, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.733 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,82e8eae095c4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131928, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116322, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.704+08:00, sub_task_instance_id=131928}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.734 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131932, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.735 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131932, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:08:01.732+08:00, endTime=null} 2023-05-15 14:08:01.736 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:08:01.737 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:08:01.737 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:08:01.737 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:08:01.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,24703dca2bc8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131929, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116323, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.719+08:00, sub_task_instance_id=131929}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.742 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d869c12e75a2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131931, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.745 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,82e8eae095c4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131928, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.746 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:08:01.746 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:08:01.746 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:08:01.746 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:08:01.746 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:08:01.747 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131933, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:01.748 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131933, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:08:01.745+08:00, endTime=null} 2023-05-15 14:08:01.749 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,24703dca2bc8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131929, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.751 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:08:01.752 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:08:01.752 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:08:01.753 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:08:01.753 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:08:01.755 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,769782cd9d00] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131932, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116326, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.732+08:00, sub_task_instance_id=131932}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.755 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,769782cd9d00] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:08:01.765 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d9231e4a9c08] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131933, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116327, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:01.745+08:00, sub_task_instance_id=131933}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:01.776 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:08:01.781 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:08:01.783 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:08:01.783 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:08:01.783 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:08:01.784 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d9231e4a9c08] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131933, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:01.785 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:08:01.785 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:08:01.797 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.797 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.797 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.798 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.798 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.798 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.798 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.798 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.799 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.799 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.799 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.799 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.799 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:08:01.800 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.801 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.801 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.801 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:08:01.801 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-3,fb27d19334114ed7,3971301d7bb2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:08:01.802 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:08:01.803 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:08:01.803 INFO 11 --- [pool-3245602-thread-2,fb27d19334114ed7,d51d6a647776] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:08:01.804 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:08:01.804 INFO 11 --- [pool-3245602-thread-1,fb27d19334114ed7,0e248f4e350b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:08:01.805 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:08:01.805 INFO 11 --- [pool-3245602-thread-4,fb27d19334114ed7,4587dbd7d98c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:08:01.806 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.807 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.807 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.807 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.808 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.808 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.808 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.808 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:01.812 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.812 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.813 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:08:01.813 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:01.813 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:01.813 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:08:01.818 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:08:01.818 INFO 11 --- [pool-3245602-thread-6,fb27d19334114ed7,3dc2d87a1c42] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:08:01.818 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:08:01.818 INFO 11 --- [pool-3245602-thread-5,fb27d19334114ed7,6bf30e976a53] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:08:01.819 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,769782cd9d00] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:08:01.828 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,769782cd9d00] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131932, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:02.744 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116322, status:SUCCESS 2023-05-15 14:08:02.745 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116322, status:SUCCESS 2023-05-15 14:08:02.753 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116323, status:SUCCESS 2023-05-15 14:08:02.755 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116323, status:SUCCESS 2023-05-15 14:08:02.765 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116324, status:SUCCESS 2023-05-15 14:08:02.766 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116324, status:SUCCESS 2023-05-15 14:08:02.774 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116325, status:SUCCESS 2023-05-15 14:08:02.775 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116325, status:SUCCESS 2023-05-15 14:08:02.786 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116326, status:SUCCESS 2023-05-15 14:08:02.787 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116326, status:SUCCESS 2023-05-15 14:08:02.798 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116327, status:SUCCESS 2023-05-15 14:08:02.800 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116327, status:SUCCESS 2023-05-15 14:08:02.845 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:02.845 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:04.714 INFO 11 --- [pool-ocp-schedules-2,,db52658634d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:08:04.721 INFO 11 --- [pool-ocp-schedules-2,,db52658634d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:08:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:04.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:04.828369+08:00, lastProcessTime=2023-05-15T14:07:54.830146+08:00 2023-05-15 14:08:04.859 INFO 11 --- [alarm-detect-14,,bcbeddeebe85] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130864 ~ 1684130874 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130870=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:04.890 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:04.828369+08:00 2023-05-15 14:08:05.846 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:05.847 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:05.901 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:06.0, nextRun=2023-05-15 14:08:05.0, currentDate=Mon May 15 14:08:05 CST 2023 2023-05-15 14:08:05.901 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:07:06.0, nextRunTime=2023-05-15 14:08:05.0} 2023-05-15 14:08:05.905 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:08:05.905 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:05.905 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:06.031 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:06.034 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:06.036 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:06.038 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:06.821 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131934, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:06.824 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131934, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:08:06.818+08:00, endTime=null} 2023-05-15 14:08:06.844 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,700db48b4faf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131934, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116328, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:06.818+08:00, sub_task_instance_id=131934}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:06.868 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,c5a308785009] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:08:06.870 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,4b33c8ceb445] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:08:06.873 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,7c1cfe919b44] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:08:06.911 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,c5a308785009] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:08:06.913 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,7c1cfe919b44] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:08:06.916 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,4b33c8ceb445] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:08:06.927 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,700db48b4faf] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131934, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:07.024 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:07.896 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116328, status:SUCCESS 2023-05-15 14:08:07.897 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116328, status:SUCCESS 2023-05-15 14:08:08.267 INFO 11 --- [pool-overview-container-3,,acaca689fcee] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:08:08.267 INFO 11 --- [pool-overview-container-3,,acaca689fcee] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:08.506 INFO 11 --- [pool-ocp-schedules-13,,196e08b5c59c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:08.520 INFO 11 --- [pool-scanner--15974117-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:08:08.847 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:08.858 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:10.370 INFO 11 --- [pool-ocp-schedules-7,,0546d1895c7a] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:08:11.858 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:11.859 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:11.895 INFO 11 --- [pool-ocp-schedules-20,,50d30a2d1ab0] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:11.895 INFO 11 --- [pool-ocp-schedules-20,,50d30a2d1ab0] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:12.330 INFO 11 --- [pool-ocp-schedules-9,,185da6870a9c] c.o.o.m.service.SeriesIdCleanService : HostIpSize=5, clusterSize=2, obproxyClusterSize=1 2023-05-15 14:08:12.414 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:14.486 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130820, nodeCount=1000, affectRows=1000 2023-05-15 14:08:14.499 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130820, nodeCount=1000, validCount=1000 2023-05-15 14:08:14.503 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130820, nodeCount=119, affectRows=119 2023-05-15 14:08:14.504 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130820, nodeCount=119, validCount=119 2023-05-15 14:08:14.509 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130820, nodeCount=1000, affectRows=1000 2023-05-15 14:08:14.513 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130820, nodeCount=1000, validCount=1000 2023-05-15 14:08:14.531 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130820, nodeCount=571, affectRows=571 2023-05-15 14:08:14.537 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130820, nodeCount=571, validCount=571 2023-05-15 14:08:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:14.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:14.828637+08:00, lastProcessTime=2023-05-15T14:08:04.828369+08:00 2023-05-15 14:08:14.859 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:14.860 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:14.861 INFO 11 --- [alarm-detect-13,,a31bfe7eeae1] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130874 ~ 1684130884 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130880=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:14.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:14.828637+08:00 2023-05-15 14:08:16.034 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:16.036 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:16.038 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:16.040 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:17.027 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:17.566 INFO 11 --- [pool-ocp-schedules-12,,2020593129d4] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:08:17.567 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:08:17.567 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:08:17.567 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:08:17.567 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:08:17.567 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:08:17.574 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:08:17.574 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:08:17.574 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:08:17.574 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:08:17.574 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:08:17.604 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:08:17.604 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:08:17.604 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:08:17.606 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:08:17.606481687+08:00 2023-05-15 14:08:17.607 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:08:17.606446105+08:00 2023-05-15 14:08:17.607 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:08:17.606462256+08:00 2023-05-15 14:08:17.607 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:08:17.606+08:00 2023-05-15 14:08:17.607 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:08:17.607+08:00 2023-05-15 14:08:17.607 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:08:17.607+08:00 2023-05-15 14:08:17.608 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:08:17.610 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:08:17.610195071+08:00 2023-05-15 14:08:17.610 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:08:17.610 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:08:17.610+08:00 2023-05-15 14:08:17.610 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:08:17.606+08:00 and status AVAILABLE 2023-05-15 14:08:17.611 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:08:17.607+08:00 and status AVAILABLE 2023-05-15 14:08:17.611 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:08:17.607+08:00 and status AVAILABLE 2023-05-15 14:08:17.613 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:08:17.612543033+08:00 2023-05-15 14:08:17.613 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:08:17.613+08:00 2023-05-15 14:08:17.614 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:08:17.610+08:00 and status AVAILABLE 2023-05-15 14:08:17.617 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:08:17.613+08:00 and status AVAILABLE 2023-05-15 14:08:17.633 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:08:17.633 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:08:17.634 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:08:17.635 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:08:17.655 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130897 0 0 , error= 2023-05-15 14:08:17.655 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130897 0 0 , error= 2023-05-15 14:08:17.655 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130897 0 0 , error= 2023-05-15 14:08:17.655 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130897 0 0 , error= 2023-05-15 14:08:17.659 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:08:17.659+08:00 2023-05-15 14:08:17.659 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:08:17.659+08:00 2023-05-15 14:08:17.659 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:08:17.659+08:00 2023-05-15 14:08:17.659 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:08:17.659+08:00 2023-05-15 14:08:17.660 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:08:17.693 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130897 0 0 , error= 2023-05-15 14:08:17.694 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:08:17.659+08:00 and status AVAILABLE 2023-05-15 14:08:17.694 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:08:17.694+08:00 2023-05-15 14:08:17.694 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:08:17.659+08:00 and status AVAILABLE 2023-05-15 14:08:17.694 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:08:17.659+08:00 and status AVAILABLE 2023-05-15 14:08:17.694 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:08:17.659+08:00 and status AVAILABLE 2023-05-15 14:08:17.695 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,b7dfe1ccbe054a2c,fdbe9c3f56d6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=b7dfe1ccbe054a2c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:08:17.697 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,b7dfe1ccbe054a2c,fdbe9c3f56d6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=b7dfe1ccbe054a2c, duration=3 ms] 2023-05-15 14:08:17.700 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:08:17.700 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:08:17.700 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:08:17.701 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:08:17.701 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:08:17.694+08:00 and status AVAILABLE 2023-05-15 14:08:17.708 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:08:17.712 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:08:17.713 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:08:17.713 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:08:17.713 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:08:17.719 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:08:17.736 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:08:17.737 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:08:17.739 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:08:17.742 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:08:17.742 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:08:17.787 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:08:17.790 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:08:17.792 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:08:17.801 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:08:17.802 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:08:17.805 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:08:17.806 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:08:17.813 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:08:17.821 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:08:17.828 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:08:17.831 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:08:17.832 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:08:17.835 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:08:17.844 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:08:17.860 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:17.861 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:17.863 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:08:17.867 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:08:17.876 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:08:17.880 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:08:17.882 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:08:17.882 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:08:17.890 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:08:17.894 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:08:17.895 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:08:17.938 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:08:17.940 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:08:17.941 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:08:17.941 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:08:17.941 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:08:17.950 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:08:17.965 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:08:17.986 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:08:17.991 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:08:17.993 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:08:18.002 INFO 11 --- [pool-Check host status22,,ebf52e8ca6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:08:18.004 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:08:18.007 INFO 11 --- [pool-Check host status9,,7b0dc3a871b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:08:18.009 INFO 11 --- [pool-Check host status26,,d115471dbf78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:08:18.015 INFO 11 --- [pool-Check host status10,,cc526062a167] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:08:18.062 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:08:18.077 INFO 11 --- [pool-Check host status12,,28b81b9146fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:08:18.108 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:08:18.110 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:08:18.111 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:08:18.111 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:08:18.111 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:08:18.115 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:08:18.116 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:08:18.121 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:08:18.146 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:08:18.148 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:08:18.150 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:08:18.150 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:08:18.151 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status25,,6f3bb55efb71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:08:18.170 INFO 11 --- [pool-Check host status18,,3391b31caa4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:08:18.179 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:08:18.179 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:08:18.179 INFO 11 --- [pool-Check host status16,,998158329081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:08:18.193 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:08:18.193 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:08:18.213 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:08:18.214 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:08:18.214 INFO 11 --- [pool-Check host status24,,0cec5e250a86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:08:18.215 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:08:18.215 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:08:18.215 INFO 11 --- [pool-Check host status15,,156a9d76350d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:08:18.266 INFO 11 --- [pool-overview-container-7,,95a82078eb30] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:18.515 INFO 11 --- [pool-scanner--15974217-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:18.528 INFO 11 --- [pool-ocp-schedules-15,,55f765cef4c4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:20.453 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:20.0, nextRun=2023-05-15 14:08:20.0, currentDate=Mon May 15 14:08:20 CST 2023 2023-05-15 14:08:20.453 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:07:20.0, nextRunTime=2023-05-15 14:08:20.0} 2023-05-15 14:08:20.456 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:08:20.456 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:20.456 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:20.861 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:20.862 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:21.006 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,7a57ab521a834499,9ea4e753492a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=7a57ab521a834499, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:21.013 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,7a57ab521a834499,9ea4e753492a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=7a57ab521a834499, duration=7 ms] 2023-05-15 14:08:21.032 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131935, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:21.035 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131935, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:08:21.030+08:00, endTime=null} 2023-05-15 14:08:21.054 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,be0565805835] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131935, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116329, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:21.030+08:00, sub_task_instance_id=131935}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:21.057 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,be0565805835] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:08:21.057 INFO 11 --- [ForkJoinPool-66050-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:08:21.057 INFO 11 --- [ForkJoinPool-66050-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:08:21.064 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,be0565805835] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131935, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:21.194 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,8ab2e054e2db42ae,9266c7670838] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=8ab2e054e2db42ae, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:21.201 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,8ab2e054e2db42ae,9266c7670838] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=8ab2e054e2db42ae, duration=7 ms] 2023-05-15 14:08:21.252 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116329, status:SUCCESS 2023-05-15 14:08:21.253 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116329, status:SUCCESS 2023-05-15 14:08:21.827 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:21.829 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:21.831 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:22.006 INFO 11 --- [pool-scanner--15974218-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:22.032 INFO 11 --- [pool-scanner--15974219-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:22.053 INFO 11 --- [pool-scanner--15974220-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:22.073 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:22.078 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:22.116 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:22.121 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:22.137 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:22.140 INFO 11 --- [pool-ocp-schedules-11,,0c530ed9e47e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:23.862 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:23.864 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:24.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:24.830928+08:00, lastProcessTime=2023-05-15T14:08:14.828637+08:00 2023-05-15 14:08:24.862 INFO 11 --- [alarm-detect-6,,7776379d981f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130884 ~ 1684130894 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130890=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:24.863 INFO 11 --- [alarm-detect-15,,142d52842b4a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130834 ~ 1684130894 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389277.988, values={1684130880=389277.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:08:24.864 INFO 11 --- [alarm-detect-15,,142d52842b4a] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130834 ~ 1684130894 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994075.358, values={1684130880=994075.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:24.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:24.830928+08:00 2023-05-15 14:08:25.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:26.0, nextRun=2023-05-15 14:08:25.0, currentDate=Mon May 15 14:08:25 CST 2023 2023-05-15 14:08:25.605 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:07:26.0, nextRunTime=2023-05-15 14:08:25.0} 2023-05-15 14:08:25.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:08:25.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:25.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:26.036 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:26.039 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:26.041 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:26.043 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:26.112 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131936, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:26.114 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131936, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:08:26.110+08:00, endTime=null} 2023-05-15 14:08:26.131 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131936, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116330, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:26.110+08:00, sub_task_instance_id=131936}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:08:26.166 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:08:26.167 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:08:26.168 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:08:26.168 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:08:26.168 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:08:26.173 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,38928487bdf0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131936, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:26.356 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116330, status:SUCCESS 2023-05-15 14:08:26.358 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116330, status:SUCCESS 2023-05-15 14:08:26.864 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:26.865 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:27.030 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:27.416 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:28.267 INFO 11 --- [pool-overview-container-1,,425f9d0f90d3] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:28.519 INFO 11 --- [pool-scanner--15974318-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:28.549 INFO 11 --- [pool-ocp-schedules-4,,2d8402123fd3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:29.865 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:29.866 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:30.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:07:31.0, nextRun=2023-05-15 14:08:30.0, currentDate=Mon May 15 14:08:30 CST 2023 2023-05-15 14:08:30.778 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:07:31.0, nextRunTime=2023-05-15 14:08:30.0} 2023-05-15 14:08:30.784 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:08:30.784 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:30.784 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:30.784 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:30.793 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:08:30.0, currentDate=Mon May 15 14:08:30 CST 2023 2023-05-15 14:08:30.793 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:08:30.0} 2023-05-15 14:08:30.796 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:08:30.796 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:08:30.796 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:08:31.185 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131938, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:31.187 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131938, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:08:31.183+08:00, endTime=null} 2023-05-15 14:08:31.196 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131939, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:31.198 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131939, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:08:31.194+08:00, endTime=null} 2023-05-15 14:08:31.210 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131938, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116331, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:31.183+08:00, sub_task_instance_id=131938}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:08:31.210 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131937, operation:EXECUTE, state: RUNNING 2023-05-15 14:08:31.212 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131937, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:08:31.208+08:00, endTime=null} 2023-05-15 14:08:31.216 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8d8d635c350e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131939, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116332, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:31.194+08:00, sub_task_instance_id=131939}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:08:31.216 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8d8d635c350e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:08:31.230 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131937, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116331, task_operation=execute, latest_execution_start_time=2023-05-15T14:08:31.208+08:00, sub_task_instance_id=131937}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:08:31.235 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:08:31.247 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.247 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.248 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.249 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.249 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.249 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.250 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.251 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:08:31.251 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.251 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.252 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-2,fb27d19334114ed7,0e6122a551ee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-1,fb27d19334114ed7,924df0dab2cf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.253 INFO 11 --- [pool-3245760-thread-3,fb27d19334114ed7,a142ce57b047] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-4,fb27d19334114ed7,89b4b6dcbeaf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.254 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.255 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.260 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.260 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.260 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.260 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.261 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:08:31.261 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:08:31.264 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:08:31.265 INFO 11 --- [pool-3245760-thread-5,fb27d19334114ed7,81e2cd876c87] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:08:31.265 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:08:31.265 INFO 11 --- [pool-3245760-thread-6,fb27d19334114ed7,e239987da058] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:08:31.266 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8d8d635c350e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:08:31.266 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.267 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.267 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.267 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.268 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.268 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.268 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:08:31.270 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8d8d635c350e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131939, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:31.270 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.271 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.271 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.272 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.275 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.275 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.276 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.276 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.276 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.276 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:08:31.279 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.279 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.280 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.288 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.288 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.289 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.291 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.291 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.292 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.301 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.301 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.304 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.305 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.305 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.318 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.318 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.319 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.320 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.320 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.320 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.328 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.328 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.329 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:08:31.333 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:08:31.340 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.340 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.341 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:08:31.350 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.350 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.350 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:08:31.359 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.359 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.359 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.360 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:08:31.360 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.361 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.361 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.362 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.367 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.367 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.368 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.370 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:08:31.385 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.385 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.386 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.386 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.386 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.386 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.386 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.387 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.387 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.387 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.387 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.387 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.388 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.392 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.393 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.398 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.398 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.399 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.399 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.399 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.400 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.400 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.400 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.423 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.423 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.423 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.424 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.425 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.429 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.429 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.430 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.439 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.442 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.442 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.442 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.450 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.452 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.452 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.452 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.454 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.459 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.459 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.460 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.461 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:08:31.465 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:08:31.467 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.468 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.468 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.468 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.469 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.470 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.470 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.471 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.484 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.484 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.485 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.485 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.485 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.486 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.486 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.486 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.491 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116332, status:SUCCESS 2023-05-15 14:08:31.493 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116332, status:SUCCESS 2023-05-15 14:08:31.501 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.502 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.503 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.503 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.504 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.508 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.508 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.508 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.508 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.509 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.510 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.510 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:08:31.511 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.524 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.525 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.525 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.525 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.525 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.526 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.527 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.527 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.527 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:08:31.527 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:08:31.529 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:08:31.530 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:31.530 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.530 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.531 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:08:31.549 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.552 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:08:31.552 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:08:31.552 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:08:31.559 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,c17a965b6c1b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131938, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:31.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:31.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.561 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.562 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.562 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:31.564 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.564 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.564 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:08:31.573 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.575 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:08:31.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:08:31.585 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.586 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:08:31.587 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.587 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.587 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:08:31.595 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.596 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:08:31.597 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:31.597 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:31.597 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:08:31.611 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:08:31.613 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:08:31.613 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:08:31.613 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:08:31.617 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a467b331c02d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131937, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:08:31.901 INFO 11 --- [pool-ocp-schedules-2,,43655389b38e] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:31.901 INFO 11 --- [pool-ocp-schedules-2,,43655389b38e] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:32.032 INFO 11 --- [pool-ocp-schedules-17,,63c58f4934b8] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:08:32.034 INFO 11 --- [pool-ocp-schedules-17,,63c58f4934b8] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:08:32.052 INFO 11 --- [pool-ocp-schedules-14,,56e894a8173f] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:08:32.053 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.053 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.053 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.054 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.054 INFO 11 --- [pool-ocp-schedules-14,,56e894a8173f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:08:32.054 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.055 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.055 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.056 INFO 11 --- [ForkJoinPool-66051-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:08:32.060 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.061 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.061 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.062 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.067 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.067 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.067 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.072 INFO 11 --- [ForkJoinPool-66051-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:08:32.076 INFO 11 --- [pool-ocp-schedules-9,,43d35dc9d9d3] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:08:32.079 INFO 11 --- [pool-ocp-schedules-9,,43d35dc9d9d3] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:08:32.108 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.109 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.109 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.113 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.120 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.120 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.121 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:08:32.129 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.130 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.131 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.131 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.136 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.136 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.137 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:08:32.181 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:32.185 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.186 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.186 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.186 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.187 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.187 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.187 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:08:32.188 WARN 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:08:32.189 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.189 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.189 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:08:32.190 INFO 11 --- [ForkJoinPool-66053-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:08:32.190 INFO 11 --- [ForkJoinPool-66053-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:08:32.190 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:08:32.201 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.201 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:08:32.214 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.223 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:32.224 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.225 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.225 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.225 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:32.225 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.225 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.226 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.226 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.226 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.227 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:08:32.227 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.227 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.230 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.230 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.230 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:08:32.233 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:08:32.233 INFO 11 --- [ForkJoinPool-66053-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:08:32.233 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.233 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.234 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:08:32.236 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:08:32.246 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.246 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:08:32.258 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.258 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:08:32.271 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.271 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:08:32.288 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.288 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:08:32.305 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.305 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:08:32.326 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:08:32.347 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:08:32.348 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.348 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.348 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.349 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.354 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:08:32.354 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:08:32.354 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:08:32.376 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:08:32.376 INFO 11 --- [ForkJoinPool-66053-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:08:32.531 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116331, status:SUCCESS 2023-05-15 14:08:32.532 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116331, status:SUCCESS 2023-05-15 14:08:32.867 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:32.868 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:33.147 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,f710ceb7ed62440a,4ec68f2884c0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=f710ceb7ed62440a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:33.153 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,f710ceb7ed62440a,4ec68f2884c0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=f710ceb7ed62440a, duration=7 ms] 2023-05-15 14:08:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:34.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:34.828814+08:00, lastProcessTime=2023-05-15T14:08:24.830928+08:00 2023-05-15 14:08:34.863 INFO 11 --- [alarm-detect-7,,8c857452ad1f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130894 ~ 1684130904 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=104.0, values={1684130900=104.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:08:34.865 INFO 11 --- [alarm-detect-7,,8c857452ad1f] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, currentState=Pending 2023-05-15 14:08:34.868 INFO 11 --- [alarm-detect-1,,635c5e3fa7ba] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130894 ~ 1684130904 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130900=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:34.872 INFO 11 --- [alarm-detect-7,,b1bdadab1128] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130844 ~ 1684130904 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389277.938, values={1684130880=389277.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:08:34.873 INFO 11 --- [alarm-detect-7,,b1bdadab1128] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130844 ~ 1684130904 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994075.107, values={1684130880=994075.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:34.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:34.828814+08:00 2023-05-15 14:08:35.288 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,28042b76f5634c84,55df51b551ad] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=28042b76f5634c84, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:35.293 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,28042b76f5634c84,55df51b551ad] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=28042b76f5634c84, duration=5 ms] 2023-05-15 14:08:35.868 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:35.869 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:36.039 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:36.041 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:36.049 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:36.051 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:37.034 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:38.267 INFO 11 --- [pool-overview-container-5,,e51db339c5fa] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:38.516 INFO 11 --- [pool-scanner--15974429-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:38.573 INFO 11 --- [pool-ocp-schedules-13,,19a5b6ffbc0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:38.869 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:38.870 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:39.212 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,30724005b1124b21,6aadc456f81e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=30724005b1124b21, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:39.217 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,30724005b1124b21,6aadc456f81e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=30724005b1124b21, duration=6 ms] 2023-05-15 14:08:41.870 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:41.872 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:42.418 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:44.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:44.830500+08:00, lastProcessTime=2023-05-15T14:08:34.828814+08:00 2023-05-15 14:08:44.863 INFO 11 --- [alarm-detect-14,,40b0d23fa674] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130894 ~ 1684130904 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=104.0, values={1684130900=104.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:08:44.867 INFO 11 --- [alarm-detect-13,,97e98cfc9d0d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130904 ~ 1684130914 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130910=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:44.872 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:44.873 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:44.905 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:44.830500+08:00 2023-05-15 14:08:45.497 INFO 11 --- [pool-ocp-schedules-13,,f1657d101c06] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:08:46.042 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:46.049 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:46.051 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:46.054 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:47.003 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001} 2023-05-15 14:08:47.032 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.032 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.034 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.034 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.036 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:47.042 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.042 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.043 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.043 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.044 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.044 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.050 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:46.931, end=2023-05-15T14:06:47.001},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:08:47.050 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:08:47.050 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:08:47.874 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:47.875 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:48.273 INFO 11 --- [pool-overview-container-2,,0aa5f9693298] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:48.514 INFO 11 --- [pool-scanner--15974529-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:48.594 INFO 11 --- [pool-ocp-schedules-12,,c895762683a5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:48.770 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,49e0889bc45947f6,ecd643bfec17] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=49e0889bc45947f6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:48.775 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,49e0889bc45947f6,ecd643bfec17] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=49e0889bc45947f6, duration=5 ms] 2023-05-15 14:08:50.875 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:50.876 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:51.732 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,dd71118963d94abc,b816c1fac822] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=dd71118963d94abc, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:08:51.734 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,dd71118963d94abc,b816c1fac822] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=dd71118963d94abc, duration=2 ms] 2023-05-15 14:08:51.830 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:51.832 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:51.834 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:51.906 INFO 11 --- [pool-ocp-schedules-16,,e9a9b0479461] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:51.906 INFO 11 --- [pool-ocp-schedules-16,,e9a9b0479461] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:08:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,86532ad50310] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:08:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,86532ad50310] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:08:53.876 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:53.877 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:08:54.835 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:08:54.834990+08:00, lastProcessTime=2023-05-15T14:08:44.830500+08:00 2023-05-15 14:08:54.848 INFO 11 --- [alarm-detect-7,,27823b8fd489] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684130894 ~ 1684130904 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=104.0, values={1684130900=104.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:08:54.859 INFO 11 --- [alarm-detect-10,,f4797b21fe96] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130914 ~ 1684130924 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130920=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:08:54.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:08:54.834990+08:00 2023-05-15 14:08:54.905 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,77ad31b8ade243aa,44c8997484c5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=77ad31b8ade243aa, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:08:54.911 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,77ad31b8ade243aa,44c8997484c5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=77ad31b8ade243aa, duration=6 ms] 2023-05-15 14:08:55.103 INFO 11 --- [pool-prometheus-second-schedule-2,,60d6c64aedc7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:08:55.103 INFO 11 --- [pool-prometheus-second-schedule-2,,dba1bf6d3470] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:08:56.045 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:56.050 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:56.052 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:56.058 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:56.877 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:56.878 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:08:57.039 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:57.420 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:08:58.267 INFO 11 --- [pool-overview-container-4,,576ffeb0adba] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:08:58.515 INFO 11 --- [pool-scanner--15974623-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:08:58.616 INFO 11 --- [pool-ocp-schedules-4,,c8d36a5515b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:08:59.879 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:08:59.881 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684130940 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-8,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:09:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:09:00.010 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:09:00.010 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:09:00.010 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:09:00.010 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:09:00.014 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:09:00.014 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:09:00.014 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:09:00.014 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:09:00.037 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:09:00.059 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:09:00.068 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:09:00.068 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:09:00.068 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:09:00.068 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:09:00.068 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:09:00.075 WARN 11 --- [pool-ocp-schedules-8,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:09:00.075 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:09:00.602 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:09:00.632 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:09:00.0, currentDate=Mon May 15 14:09:00 CST 2023 2023-05-15 14:09:00.632 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:09:00.0} 2023-05-15 14:09:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:09:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:00.646 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:09:00.0, currentDate=Mon May 15 14:09:00 CST 2023 2023-05-15 14:09:00.647 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:09:00.0} 2023-05-15 14:09:00.657 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:09:00.658 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:00.658 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:00.663 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:09:00.0, currentDate=Mon May 15 14:09:00 CST 2023 2023-05-15 14:09:00.663 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:09:00.0} 2023-05-15 14:09:00.674 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:09:00.674 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:00.674 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:00.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:31.0, nextRun=2023-05-15 14:09:00.0, currentDate=Mon May 15 14:09:00 CST 2023 2023-05-15 14:09:00.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:08:31.0, nextRunTime=2023-05-15 14:09:00.0} 2023-05-15 14:09:00.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:09:00.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:00.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:09:00.0, currentDate=Mon May 15 14:09:00 CST 2023 2023-05-15 14:09:00.690 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:09:00.0} 2023-05-15 14:09:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:09:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:01.084 WARN 11 --- [pool-ocp-schedules-8,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:09:01.086 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.071221591S 2023-05-15 14:09:01.086 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:09:01.213 INFO 11 --- [pool-ocp-schedules-8,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:09:01.285 INFO 11 --- [pool-ocp-schedules-8,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:09:01.327 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:09:01.327 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:09:01.327 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:09:01.327 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:09:01.327 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:09:01.328 INFO 11 --- [pool-ocp-schedules-8,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:09:01.339 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:09:01.484 INFO 11 --- [pool-ocp-schedules-8,,] o.s.boot.SpringApplication : Started application in 1.475 seconds (JVM running for 600597.458) 2023-05-15 14:09:01.485 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.475184146S 2023-05-15 14:09:01.513 INFO 11 --- [pool-prometheus-minute-schedule-1,,476bcbd46cae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131000, interval=60 2023-05-15 14:09:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,cd4f82cbca24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131000, interval=60 2023-05-15 14:09:01.551 INFO 11 --- [pool-prometheus-minute-schedule-2,,cd4f82cbca24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:09:01.551 INFO 11 --- [pool-prometheus-minute-schedule-2,,cd4f82cbca24] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130941, count=6, elapsed=1682446810610 2023-05-15 14:09:01.555 INFO 11 --- [pool-prometheus-minute-schedule-1,,476bcbd46cae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:09:01.555 INFO 11 --- [pool-prometheus-minute-schedule-1,,476bcbd46cae] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684130941, count=8, elapsed=1682446810614 2023-05-15 14:09:01.625 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131942, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:01.627 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131942, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:09:01.618+08:00, endTime=null} 2023-05-15 14:09:01.643 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131943, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:01.645 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131943, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:09:01.640+08:00, endTime=null} 2023-05-15 14:09:01.646 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,d9a7661166d7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131942, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116335, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:01.615+08:00, sub_task_instance_id=131942}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:01.653 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131940, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:01.655 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131940, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:09:01.651+08:00, endTime=null} 2023-05-15 14:09:01.662 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,200329169ce8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131943, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116336, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:01.634+08:00, sub_task_instance_id=131943}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:01.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,d9a7661166d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:09:01.662 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,200329169ce8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:09:01.662 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,d9a7661166d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:09:01.662 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131941, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:01.663 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131941, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:09:01.660+08:00, endTime=null} 2023-05-15 14:09:01.664 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,d9a7661166d7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:09:01.669 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,d9a7661166d7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131942, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:01.669 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1959bf11a274] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131940, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116333, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:01.651+08:00, sub_task_instance_id=131940}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:01.672 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131944, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:01.674 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131944, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:09:01.670+08:00, endTime=null} 2023-05-15 14:09:01.676 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:09:01.677 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:09:01.677 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:09:01.677 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:09:01.677 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:09:01.680 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:09:01.680 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:09:01.680 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,35e269a3fc94] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131941, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116334, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:01.660+08:00, sub_task_instance_id=131941}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:01.680 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:09:01.680 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:09:01.681 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:09:01.681 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:09:01.681 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:09:01.681 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:09:01.682 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1959bf11a274] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131940, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:01.682 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:09:01.683 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:09:01.684 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:09:01.692 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.693 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,5003909bef75] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131944, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116337, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:01.670+08:00, sub_task_instance_id=131944}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.693 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.695 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.695 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.695 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.696 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.698 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.699 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.699 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.699 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.700 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.701 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.701 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.701 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,35e269a3fc94] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:09:01.701 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,35e269a3fc94] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:09:01.702 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:01.706 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.706 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.706 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:09:01.707 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:09:01.707 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,35e269a3fc94] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131941, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:01.708 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,5003909bef75] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131944, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:01.708 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:09:01.709 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:09:01.709 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:09:01.710 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.710 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.710 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:09:01.710 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:09:01.710 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:09:01.711 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:09:01.712 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:09:01.713 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.713 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.713 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:09:01.715 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:01.715 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.715 INFO 11 --- [pool-3245929-thread-4,fb27d19334114ed7,19c90cc25b4f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:09:01.715 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:09:01.716 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:09:01.716 INFO 11 --- [pool-3245929-thread-3,fb27d19334114ed7,728249f91329] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:09:01.717 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:09:01.718 INFO 11 --- [pool-3245929-thread-1,fb27d19334114ed7,f9b73778ce55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:09:01.718 INFO 11 --- [pool-3245929-thread-2,fb27d19334114ed7,234b1e437a3a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:09:01.719 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.719 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.719 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:09:01.720 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:09:01.720 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:09:01.720 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:01.720 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:01.721 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:09:01.725 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:09:01.725 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:09:01.725 INFO 11 --- [pool-3245929-thread-5,fb27d19334114ed7,8fa444ae7f9b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:09:01.725 INFO 11 --- [pool-3245929-thread-6,fb27d19334114ed7,97ad065c6ecb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:09:01.726 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,200329169ce8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:09:01.727 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:09:01.730 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,200329169ce8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131943, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:01.735 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:09:01.736 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:09:01.736 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:09:01.736 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:09:01.736 INFO 11 --- [pool-ocp-schedules-8,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:09:01.738 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 3 2023-05-15 14:09:01.744 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.s.a.service.OcpAlarmServiceImpl : delete Done, affectRow=1, id=495 2023-05-15 14:09:01.747 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event Done, affectRow=1, id=496 2023-05-15 14:09:01.750 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.s.a.service.OcpAlarmServiceImpl : create alarm event history Done, affectRow=1, id=496 2023-05-15 14:09:01.762 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm rootservice_log_alarm null 2023-05-15 14:09:01.769 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:09:01.776 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:09:01.994 INFO 11 --- [pool-ocp-schedules-8,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:09:02.122 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116333, status:SUCCESS 2023-05-15 14:09:02.124 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116333, status:SUCCESS 2023-05-15 14:09:02.138 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116334, status:SUCCESS 2023-05-15 14:09:02.139 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116334, status:SUCCESS 2023-05-15 14:09:02.153 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116335, status:SUCCESS 2023-05-15 14:09:02.154 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116335, status:SUCCESS 2023-05-15 14:09:02.166 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116336, status:SUCCESS 2023-05-15 14:09:02.167 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116336, status:SUCCESS 2023-05-15 14:09:02.180 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116337, status:SUCCESS 2023-05-15 14:09:02.180 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116337, status:SUCCESS 2023-05-15 14:09:02.881 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:02.882 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:04.713 INFO 11 --- [pool-ocp-schedules-19,,6fb02e2ece54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:09:04.720 INFO 11 --- [pool-ocp-schedules-19,,6fb02e2ece54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:09:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:04.830809+08:00, lastProcessTime=2023-05-15T14:08:54.834990+08:00 2023-05-15 14:09:04.867 INFO 11 --- [alarm-detect-2,,5d765bb1367e] c.o.o.s.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, activeAt=2023-05-15T14:08:34.852+08:00 2023-05-15 14:09:04.868 INFO 11 --- [alarm-detect-2,,5d765bb1367e] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType tenant_cpu_percent_over_threshold targets.size 10 was marked as not match 2023-05-15 14:09:04.870 INFO 11 --- [alarm-detect-11,,6b8996d98011] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130924 ~ 1684130934 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130930=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:04.904 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:04.830809+08:00 2023-05-15 14:09:05.838 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:06.0, nextRun=2023-05-15 14:09:05.0, currentDate=Mon May 15 14:09:05 CST 2023 2023-05-15 14:09:05.838 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:08:06.0, nextRunTime=2023-05-15 14:09:05.0} 2023-05-15 14:09:05.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:09:05.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:05.843 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:05.883 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:05.883 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:06.048 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:06.052 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:06.059 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:06.063 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:06.757 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131945, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:06.760 INFO 11 --- [pool-subtask-coordinator1,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131945, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:09:06.755+08:00, endTime=null} 2023-05-15 14:09:06.776 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f1117cc27686] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131945, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116338, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:06.755+08:00, sub_task_instance_id=131945}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:06.796 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,7700a08fd40c] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:09:06.799 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,f6aaea6bc97e] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:09:06.803 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,582c6410eae2] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:09:06.838 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,7700a08fd40c] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:09:06.850 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,f6aaea6bc97e] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:09:06.854 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,582c6410eae2] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:09:06.873 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,f1117cc27686] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131945, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:07.042 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:07.287 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116338, status:SUCCESS 2023-05-15 14:09:07.289 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116338, status:SUCCESS 2023-05-15 14:09:08.270 INFO 11 --- [pool-overview-container-8,,624a5610a714] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=6 2023-05-15 14:09:08.270 INFO 11 --- [pool-overview-container-8,,624a5610a714] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=495 2023-05-15 14:09:08.524 INFO 11 --- [pool-scanner--15974716-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:09:08.568 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5ad3b836cffd44a6,2ef299c4e8e6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=5ad3b836cffd44a6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:08.574 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,5ad3b836cffd44a6,2ef299c4e8e6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=5ad3b836cffd44a6, duration=7 ms] 2023-05-15 14:09:08.636 INFO 11 --- [pool-ocp-schedules-5,,2201d51845f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:08.884 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:08.885 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:09.184 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,81668d76ce384b76,b8801d3c23d4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=81668d76ce384b76, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:09.188 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,81668d76ce384b76,b8801d3c23d4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=81668d76ce384b76, duration=4 ms] 2023-05-15 14:09:09.948 INFO 11 --- [pool-ocp-schedules-8,,5b997a5b9db2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:09:09.948 INFO 11 --- [pool-ocp-schedules-8,,5b997a5b9db2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:09:10.409 INFO 11 --- [pool-ocp-schedules-6,,e2a56c29ef94] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=6 2023-05-15 14:09:11.885 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:11.886 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:11.911 INFO 11 --- [pool-ocp-schedules-15,,c850e99117f3] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:11.911 INFO 11 --- [pool-ocp-schedules-15,,c850e99117f3] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:12.428 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:14.537 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130880, nodeCount=690, affectRows=690 2023-05-15 14:09:14.539 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130880, nodeCount=1000, affectRows=1000 2023-05-15 14:09:14.543 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130880, nodeCount=1000, validCount=1000 2023-05-15 14:09:14.543 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130880, nodeCount=690, validCount=690 2023-05-15 14:09:14.544 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130880, nodeCount=1000, affectRows=1000 2023-05-15 14:09:14.548 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130880, nodeCount=1000, validCount=1000 2023-05-15 14:09:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:14.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:14.827537+08:00, lastProcessTime=2023-05-15T14:09:04.830809+08:00 2023-05-15 14:09:14.859 INFO 11 --- [alarm-detect-8,,15193c9586ae] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130934 ~ 1684130944 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130940=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:14.886 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:14.887 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:14.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:14.827537+08:00 2023-05-15 14:09:16.051 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:16.059 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:16.063 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:16.065 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:17.046 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:17.620 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:09:17.665 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:17.665 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:17.665 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.666 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:17.675 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:17.675 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.675 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:09:17.693 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:17.695 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:17.696 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:17.696 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.697 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:17.703 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:17.703 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.704 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:09:17.706 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:09:17.706 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:09:17.706 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:09:17.706 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:09:17.711 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:09:17.711 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.712 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:09:17.712 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.712 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:09:17.712 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:17.713 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:09:17.713 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:09:17.713 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:09:17.736 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:06:48, start=2023-05-15T14:03:17, end=2023-05-15T14:05:47, updates=91, granule=120000ms, delay=211000ms 2023-05-15 14:09:17.737 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:07:02, start=2023-05-15T14:03:17, end=2023-05-15T14:05:47, updates=108, granule=120000ms, delay=225000ms 2023-05-15 14:09:17.737 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:06:48, target=2023-05-15T14:05:47, granule=120000ms, delay=61000ms) 2023-05-15 14:09:17.737 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:05:47, target=2023-05-15T14:03:17, granule=600000ms, delay=150000ms) 2023-05-15 14:09:17.739 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:07:02, target=2023-05-15T14:05:47, granule=120000ms, delay=75000ms) 2023-05-15 14:09:17.745 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:06:48, start=2023-05-15T14:03:17, end=2023-05-15T14:05:47, updates=133, granule=120000ms, delay=211000ms 2023-05-15 14:09:17.750 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:06:48, target=2023-05-15T14:05:47, granule=120000ms, delay=61000ms) 2023-05-15 14:09:17.750 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:05:47, target=2023-05-15T14:03:17, granule=600000ms, delay=150000ms) 2023-05-15 14:09:17.830 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=1->2, c=0) - source=2023-05-15T14:05:47, start=2023-05-15T13:53:17, end=2023-05-15T14:05:17, updates=133, granule=600000ms, delay=750000ms 2023-05-15 14:09:17.831 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=1) - not enough cached interval (source=2023-05-15T14:05:47, target=2023-05-15T14:05:17, granule=600000ms, delay=30000ms) 2023-05-15 14:09:17.831 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=211ms, period=120000ms 2023-05-15 14:09:17.887 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:17.888 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:18.227 INFO 11 --- [pool-ocp-schedules-13,,9734ee0e3c9f] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:09:18.227 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:09:18.227 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:09:18.227 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:09:18.227 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:09:18.227 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:09:18.233 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:09:18.233 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:09:18.233 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:09:18.234 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:09:18.237 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:09:18.263 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:09:18.264 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:09:18.265 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:09:18.265266123+08:00 2023-05-15 14:09:18.266 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:09:18.265+08:00 2023-05-15 14:09:18.266 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:09:18.266451996+08:00 2023-05-15 14:09:18.267 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:09:18.266+08:00 2023-05-15 14:09:18.267 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:09:18.268 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:09:18.270 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:09:18.269533268+08:00 2023-05-15 14:09:18.270 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:09:18.269992379+08:00 2023-05-15 14:09:18.271 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:09:18.270+08:00 2023-05-15 14:09:18.271 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:09:18.270+08:00 2023-05-15 14:09:18.277 INFO 11 --- [pool-overview-container-6,,3faa22fe18b6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:18.279 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:09:18.265+08:00 and status AVAILABLE 2023-05-15 14:09:18.280 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:09:18.270+08:00 and status AVAILABLE 2023-05-15 14:09:18.283 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:09:18.266+08:00 and status AVAILABLE 2023-05-15 14:09:18.285 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:09:18.270+08:00 and status AVAILABLE 2023-05-15 14:09:18.287 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:09:18.289 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:09:18.292 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:09:18.311 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130958 0 0 , error= 2023-05-15 14:09:18.311 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:09:18.311+08:00 2023-05-15 14:09:18.321 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130958 0 0 , error= 2023-05-15 14:09:18.325 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:09:18.321+08:00 2023-05-15 14:09:18.329 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:09:18.321+08:00 and status AVAILABLE 2023-05-15 14:09:18.336 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:09:18.311+08:00 and status AVAILABLE 2023-05-15 14:09:18.336 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:09:18.336 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130958 0 0 , error= 2023-05-15 14:09:18.337 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:09:18.337+08:00 2023-05-15 14:09:18.337 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:09:18.339 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:09:18.338977204+08:00 2023-05-15 14:09:18.339 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:09:18.339+08:00 2023-05-15 14:09:18.352 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:09:18.352 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:09:18.352 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:09:18.337+08:00 and status AVAILABLE 2023-05-15 14:09:18.354 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:09:18.339+08:00 and status AVAILABLE 2023-05-15 14:09:18.355 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:09:18.361 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:09:18.361 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:09:18.361 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:09:18.363 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:09:18.394 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130958 0 0 , error= 2023-05-15 14:09:18.394 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:09:18.394+08:00 2023-05-15 14:09:18.399 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:09:18.394+08:00 and status AVAILABLE 2023-05-15 14:09:18.402 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:09:18.413 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:09:18.418 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:09:18.419 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:09:18.431 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:09:18.442 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:09:18.468 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:09:18.469 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:09:18.483 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:09:18.484 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:09:18.494 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:09:18.503 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:09:18.507 INFO 11 --- [pool-scanner--15974816-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:18.512 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:09:18.513 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:09:18.514 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:09:18.531 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:09:18.542 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:09:18.559 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:09:18.563 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:09:18.564 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:09:18.565 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:09:18.578 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:09:18.579 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:09:18.579 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:09:18.599 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:09:18.599 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:09:18.599 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:09:18.646 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:09:18.647 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:09:18.650 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:09:18.660 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:09:18.660 INFO 11 --- [pool-ocp-schedules-2,,bdc2119a318f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:18.661 INFO 11 --- [pool-Check host status6,,fc2f57b9c3c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:09:18.663 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:09:18.665 INFO 11 --- [pool-Check host status4,,c8d3994214e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:09:18.675 INFO 11 --- [pool-Check host status7,,7445a3e85d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:09:18.678 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:09:18.769 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:09:18.782 INFO 11 --- [pool-Check host status21,,f59610c51b5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:09:19.297 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684130959 0 0 , error= 2023-05-15 14:09:19.298 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:09:19.297+08:00 2023-05-15 14:09:19.302 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:09:19.297+08:00 and status AVAILABLE 2023-05-15 14:09:19.305 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:09:19.317 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:09:19.351 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:09:19.408 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:09:19.423 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:09:19.444 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:09:19.488 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:09:19.504 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:09:19.527 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:09:19.587 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:09:19.601 INFO 11 --- [pool-Check host status29,,05505a07bacf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:09:19.622 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:09:19.622 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:09:19.622 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:09:19.622 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:09:19.623 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:09:19.629 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:09:19.629 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:09:19.629 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:09:19.630 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:09:19.630 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:09:19.733 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:09:19.733 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:09:19.733 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:09:19.735 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:09:19.735 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:09:19.753 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:09:19.753 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:09:19.753 INFO 11 --- [pool-Check host status23,,5f29a57c25c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:09:19.753 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:09:19.754 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:09:19.754 INFO 11 --- [pool-Check host status20,,0a315a7bcfe7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:09:19.754 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:09:19.754 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:09:19.754 INFO 11 --- [pool-Check host status5,,57e54b06d34b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:09:19.756 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:09:19.756 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:09:19.756 INFO 11 --- [pool-Check host status27,,6a515adfb82a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:09:19.760 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:09:19.760 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:09:19.760 INFO 11 --- [pool-Check host status11,,b30c966b6a4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:09:20.286 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:20.0, nextRun=2023-05-15 14:09:20.0, currentDate=Mon May 15 14:09:20 CST 2023 2023-05-15 14:09:20.286 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:08:20.0, nextRunTime=2023-05-15 14:09:20.0} 2023-05-15 14:09:20.291 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:09:20.291 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:20.291 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:20.888 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:20.889 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:20.961 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131946, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:20.963 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131946, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:09:20.958+08:00, endTime=null} 2023-05-15 14:09:20.985 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,85dd59960022] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131946, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116339, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:20.958+08:00, sub_task_instance_id=131946}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:20.988 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,85dd59960022] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:09:20.989 INFO 11 --- [ForkJoinPool-66056-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:09:20.989 INFO 11 --- [ForkJoinPool-66056-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:09:20.993 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,85dd59960022] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131946, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:21.623 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116339, status:SUCCESS 2023-05-15 14:09:21.625 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116339, status:SUCCESS 2023-05-15 14:09:21.832 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:21.834 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:21.836 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:22.179 INFO 11 --- [pool-scanner--15974817-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:22.202 INFO 11 --- [pool-scanner--15974818-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:22.224 INFO 11 --- [pool-scanner--15974819-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:22.249 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:22.254 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:22.295 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:22.299 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:22.317 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:22.321 INFO 11 --- [pool-ocp-schedules-11,,e0cd9496e492] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:23.890 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:23.891 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:24.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:24.832485+08:00, lastProcessTime=2023-05-15T14:09:14.827537+08:00 2023-05-15 14:09:24.864 INFO 11 --- [alarm-detect-12,,e4621aae1fd8] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130944 ~ 1684130954 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130950=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:24.866 INFO 11 --- [alarm-detect-1,,7195fc90ac69] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130894 ~ 1684130954 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389337.988, values={1684130940=389337.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:09:24.868 INFO 11 --- [alarm-detect-1,,7195fc90ac69] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130894 ~ 1684130954 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994135.358, values={1684130940=994135.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:24.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:24.832485+08:00 2023-05-15 14:09:25.440 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:26.0, nextRun=2023-05-15 14:09:25.0, currentDate=Mon May 15 14:09:25 CST 2023 2023-05-15 14:09:25.440 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:08:26.0, nextRunTime=2023-05-15 14:09:25.0} 2023-05-15 14:09:25.446 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:09:25.446 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:25.446 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:25.766 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,87c00c22b616465f,36283609a4a3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=87c00c22b616465f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:09:25.768 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,87c00c22b616465f,36283609a4a3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=87c00c22b616465f, duration=3 ms] 2023-05-15 14:09:26.056 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131947, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:26.059 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:26.059 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131947, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:09:26.052+08:00, endTime=null} 2023-05-15 14:09:26.062 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:26.068 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:26.090 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:26.101 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131947, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116340, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:26.052+08:00, sub_task_instance_id=131947}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:09:26.131 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:09:26.132 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:09:26.132 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:09:26.133 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:09:26.133 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:09:26.137 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,d71a603b4d99] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131947, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:26.582 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,5a87821cc501463f,a1a181354aa4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=5a87821cc501463f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:26.588 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,5a87821cc501463f,a1a181354aa4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=5a87821cc501463f, duration=6 ms] 2023-05-15 14:09:26.731 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116340, status:SUCCESS 2023-05-15 14:09:26.732 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116340, status:SUCCESS 2023-05-15 14:09:26.891 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:26.892 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:27.049 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:27.431 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:28.268 INFO 11 --- [pool-overview-container-2,,50a4c4c8e66d] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:28.518 INFO 11 --- [pool-scanner--15974917-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:28.679 INFO 11 --- [pool-ocp-schedules-17,,72e53fdcc048] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:28.803 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,5a0d213c26644413,6d572d04a5ee] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=5a0d213c26644413, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:28.809 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,5a0d213c26644413,6d572d04a5ee] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=5a0d213c26644413, duration=6 ms] 2023-05-15 14:09:29.875 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b5b17ac7787149db,ff59967f8524] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=b5b17ac7787149db, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:29.892 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:29.893 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b5b17ac7787149db,ff59967f8524] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=b5b17ac7787149db, duration=18 ms] 2023-05-15 14:09:29.894 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:30.591 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/parameters?null, client=192.168.80.229, traceId=0cfcbb8bf93d4c6f, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantParameterController.listTenantParameters(Long,Long,Boolean,Boolean), args=2,6,] 2023-05-15 14:09:30.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:31.0, nextRun=2023-05-15 14:09:30.0, currentDate=Mon May 15 14:09:30 CST 2023 2023-05-15 14:09:30.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:08:31.0, nextRunTime=2023-05-15 14:09:30.0} 2023-05-15 14:09:30.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:09:30.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:30.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:30.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:30.620 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:09:30.621 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:09:30.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:01.0, nextRun=2023-05-15 14:09:30.0, currentDate=Mon May 15 14:09:30 CST 2023 2023-05-15 14:09:30.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:09:01.0, nextRunTime=2023-05-15 14:09:30.0} 2023-05-15 14:09:30.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:09:30.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:09:30.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:09:30.633 ERROR 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573908) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573908) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:09:30.634 ERROR 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] com.alibaba.druid.pool.DruidDataSource : {dataSource-101} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573908) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573908) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:09:30.635 INFO 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] com.alibaba.druid.pool.DruidDataSource : {dataSource-101,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:09:30.635 INFO 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] com.alibaba.druid.pool.DruidDataSource : {dataSource-101} closing ... 2023-05-15 14:09:30.635 INFO 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] com.alibaba.druid.pool.DruidDataSource : {dataSource-101} closed 2023-05-15 14:09:30.635 ERROR 11 --- [obsdk-init-thread-2,0cfcbb8bf93d4c6f,eb1483e9d40b] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573908) Get Location Cache Fail 2023-05-15 14:09:30.639 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573908) Get Location Cache Fail 2023-05-15 14:09:30.639 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573908) Get Location Cache Fail 2023-05-15 14:09:30.640 WARN 11 --- [http-nio-0.0.0.0-8080-exec-66,0cfcbb8bf93d4c6f,a9422624dbf3] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/parameters client=192.168.80.229, traceId=0cfcbb8bf93d4c6f, duration=52 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:09:30.646 ERROR 11 --- [Druid-ConnectionPool-Create-370172475,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573911) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573911) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:09:31.141 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131950, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:31.143 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131950, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:09:31.139+08:00, endTime=null} 2023-05-15 14:09:31.155 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131948, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:31.159 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131948, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:09:31.152+08:00, endTime=null} 2023-05-15 14:09:31.163 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ab5020a9caf7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131950, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116342, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:31.138+08:00, sub_task_instance_id=131950}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:09:31.163 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ab5020a9caf7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:09:31.169 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131949, operation:EXECUTE, state: RUNNING 2023-05-15 14:09:31.171 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131949, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:09:31.167+08:00, endTime=null} 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:09:31.182 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:09:31.184 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131948, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116341, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:31.152+08:00, sub_task_instance_id=131948}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:09:31.193 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.193 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131949, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116341, task_operation=execute, latest_execution_start_time=2023-05-15T14:09:31.167+08:00, sub_task_instance_id=131949}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:09:31.193 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.193 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.193 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.194 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.195 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.196 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.196 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:09:31.197 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.198 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:09:31.198 INFO 11 --- [pool-3246087-thread-4,fb27d19334114ed7,4f3e1a5a42f5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.199 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.200 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:09:31.200 INFO 11 --- [pool-3246087-thread-3,fb27d19334114ed7,1acf100445bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:09:31.200 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.200 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:09:31.201 INFO 11 --- [pool-3246087-thread-2,fb27d19334114ed7,65b10dd957c8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:09:31.201 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.201 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.201 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.202 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:09:31.202 INFO 11 --- [pool-3246087-thread-1,fb27d19334114ed7,b6831f58f676] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:09:31.202 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.205 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.205 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.206 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:09:31.206 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.206 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.207 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:09:31.211 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:09:31.211 INFO 11 --- [pool-3246087-thread-5,fb27d19334114ed7,f6c7961e4493] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:09:31.212 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:09:31.212 INFO 11 --- [pool-3246087-thread-6,fb27d19334114ed7,d8b5e56aa980] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:09:31.213 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ab5020a9caf7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:09:31.217 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,ab5020a9caf7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131950, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:31.220 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.220 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.220 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.220 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.227 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.227 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.227 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:09:31.237 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.238 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.238 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.243 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.243 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.244 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.244 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.247 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.248 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.248 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.248 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.249 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:09:31.257 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.257 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.258 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.264 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.271 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.271 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.272 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.274 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.274 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.274 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.285 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.285 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.285 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.291 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:09:31.301 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.301 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.302 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.302 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:09:31.314 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:09:31.314 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.314 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.315 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:09:31.325 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.325 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.326 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:09:31.337 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.338 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.339 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.339 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.339 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.343 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:09:31.349 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.349 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.349 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.360 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.361 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.361 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.361 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.361 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.362 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.362 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.363 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.364 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.365 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.365 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.365 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.366 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.373 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.374 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.374 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.374 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.374 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.375 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.375 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.375 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.376 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.376 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.376 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.393 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.394 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.395 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.395 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.395 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.401 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.402 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.403 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.403 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.403 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.405 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.406 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.406 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.407 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.408 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.408 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.408 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.425 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.425 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.426 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.426 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.427 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.427 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.427 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.428 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.428 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.428 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.428 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.432 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.432 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.433 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.433 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:09:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.440 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.441 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.441 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.442 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.442 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:09:31.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.456 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.457 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.458 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:09:31.464 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.465 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.465 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.465 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.466 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.467 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.467 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.467 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.467 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.468 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.469 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.469 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.469 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:09:31.470 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:09:31.471 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.471 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.472 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.473 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:09:31.482 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.485 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:09:31.486 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:31.486 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.487 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.488 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:31.488 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:31.491 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.491 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.491 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:09:31.501 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.504 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:09:31.504 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:09:31.504 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:09:31.504 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:09:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.507 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:09:31.508 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bf07f28a313a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131949, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:31.517 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:09:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.519 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:09:31.530 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.532 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:09:31.533 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:31.533 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:31.533 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:09:31.542 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:09:31.545 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:09:31.545 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:09:31.545 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:09:31.550 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,103b5b8baecd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131948, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:09:31.836 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116341, status:SUCCESS 2023-05-15 14:09:31.838 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116341, status:SUCCESS 2023-05-15 14:09:31.851 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116342, status:SUCCESS 2023-05-15 14:09:31.853 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116342, status:SUCCESS 2023-05-15 14:09:31.916 INFO 11 --- [pool-ocp-schedules-2,,183abfe8c154] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:31.917 INFO 11 --- [pool-ocp-schedules-2,,183abfe8c154] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:32.040 INFO 11 --- [pool-ocp-schedules-12,,20f4af7f923e] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:09:32.042 INFO 11 --- [pool-ocp-schedules-12,,20f4af7f923e] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:09:32.059 INFO 11 --- [pool-ocp-schedules-14,,609e9c7843fd] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:09:32.061 INFO 11 --- [pool-ocp-schedules-14,,609e9c7843fd] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:09:32.068 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.068 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.068 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.069 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.070 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.070 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.070 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.071 INFO 11 --- [ForkJoinPool-66057-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:09:32.076 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.077 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.077 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.078 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.082 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.082 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.083 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.084 INFO 11 --- [pool-ocp-schedules-15,,3ff147d525d7] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:09:32.086 INFO 11 --- [pool-ocp-schedules-15,,3ff147d525d7] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:09:32.086 INFO 11 --- [ForkJoinPool-66057-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:09:32.113 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.114 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.114 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.114 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.129 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.130 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.130 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:09:32.132 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.133 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.133 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.134 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.138 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.138 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.139 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:09:32.178 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:32.178 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.179 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.179 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.179 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.181 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.181 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.181 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:09:32.183 WARN 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:09:32.183 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.183 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.184 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:09:32.185 INFO 11 --- [ForkJoinPool-66059-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:09:32.185 INFO 11 --- [ForkJoinPool-66059-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:09:32.185 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:09:32.203 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.203 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:09:32.220 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:32.221 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.221 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.221 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.222 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.226 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.226 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.227 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:09:32.229 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.229 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.229 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.230 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:09:32.232 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:09:32.246 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.247 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:09:32.246 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:32.248 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.248 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.249 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.249 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.250 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:09:32.250 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.251 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.260 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:09:32.260 INFO 11 --- [ForkJoinPool-66059-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:09:32.270 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.270 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:09:32.285 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.285 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:09:32.304 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.304 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:09:32.318 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.318 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:09:32.333 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:09:32.354 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:09:32.355 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.355 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.355 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.356 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.360 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:09:32.360 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:09:32.361 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:09:32.377 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:09:32.377 INFO 11 --- [ForkJoinPool-66059-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:09:32.894 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:32.895 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:34.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:34.831816+08:00, lastProcessTime=2023-05-15T14:09:24.832485+08:00 2023-05-15 14:09:34.856 INFO 11 --- [alarm-detect-11,,3ebdcb3478a2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130954 ~ 1684130964 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130960=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:34.868 INFO 11 --- [alarm-detect-14,,fa9c577afb93] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130904 ~ 1684130964 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389337.938, values={1684130940=389337.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:09:34.869 INFO 11 --- [alarm-detect-14,,fa9c577afb93] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130904 ~ 1684130964 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994135.107, values={1684130940=994135.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:34.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:34.831816+08:00 2023-05-15 14:09:35.895 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:35.896 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:36.061 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:36.068 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:36.090 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:36.093 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:37.052 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:38.268 INFO 11 --- [pool-overview-container-4,,e0a99684197e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:38.516 INFO 11 --- [pool-scanner--15975028-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:38.701 INFO 11 --- [pool-ocp-schedules-7,,fdf6b918ad28] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:38.897 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:38.898 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:41.898 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:41.899 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:42.433 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:43.081 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a846fa4399c34f69,3b362765264e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=a846fa4399c34f69, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:43.086 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a846fa4399c34f69,3b362765264e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=a846fa4399c34f69, duration=6 ms] 2023-05-15 14:09:44.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:44.831689+08:00, lastProcessTime=2023-05-15T14:09:34.831816+08:00 2023-05-15 14:09:44.857 INFO 11 --- [alarm-detect-11,,c9046f6757b2] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130964 ~ 1684130974 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130970=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:44.860 INFO 11 --- [alarm-detect-11,,c9046f6757b2] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:09:44.899 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:44.900 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:44.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:44.831689+08:00 2023-05-15 14:09:45.493 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,9d56cd1d5c644f75,562dc6b18496] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=9d56cd1d5c644f75, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:45.498 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,9d56cd1d5c644f75,562dc6b18496] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=9d56cd1d5c644f75, duration=6 ms] 2023-05-15 14:09:45.498 INFO 11 --- [pool-ocp-schedules-7,,a7eaa7f7d848] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:09:46.066 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:46.104 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:46.118 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:46.120 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:47.056 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:47.900 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:47.901 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:48.269 INFO 11 --- [pool-overview-container-8,,271c75368173] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:09:48.269 INFO 11 --- [pool-overview-container-8,,271c75368173] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:48.518 INFO 11 --- [pool-scanner--15975128-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:48.723 INFO 11 --- [pool-ocp-schedules-3,,f9a5cee773a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:50.430 INFO 11 --- [pool-ocp-schedules-8,,8f4d5aeda2fc] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:09:50.902 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:50.904 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:51.834 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:51.837 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:51.838 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:51.922 INFO 11 --- [pool-ocp-schedules-14,,8fede1cbd633] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:51.922 INFO 11 --- [pool-ocp-schedules-14,,8fede1cbd633] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:09:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,da0f36fb99c0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40057} 2023-05-15 14:09:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,da0f36fb99c0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:09:53.828 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.870 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.876 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.880 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.886 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.886 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:53.888 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.889 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.891 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.892 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.892 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:53.894 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.895 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.897 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.898 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.898 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:53.905 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:53.906 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:53.911 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.922 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 8 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.938 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 26 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.973 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 126 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:53.997 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:53.999 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.001 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.002 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.004 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.004 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:54.005 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.007 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.009 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.011 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:03:53.617, end=2023-05-15T14:08:53.821} 2023-05-15 14:09:54.011 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:09:54.011 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:09:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:09:54.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:09:54.833468+08:00, lastProcessTime=2023-05-15T14:09:44.831689+08:00 2023-05-15 14:09:54.858 INFO 11 --- [alarm-detect-1,,bb31e8a272ae] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130974 ~ 1684130984 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=245.994, values={1684130980=245.994}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:09:54.889 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:09:54.833468+08:00 2023-05-15 14:09:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,015d70aac282] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:09:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,09f274fb89f6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:09:55.941 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,97ba54cc48ff4413,00cb5af66b2b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=97ba54cc48ff4413, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:09:55.947 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,97ba54cc48ff4413,00cb5af66b2b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=97ba54cc48ff4413, duration=6 ms] 2023-05-15 14:09:56.104 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:56.118 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:56.120 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:56.122 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:56.906 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:56.907 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:09:57.059 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:57.436 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:09:58.268 INFO 11 --- [pool-overview-container-6,,7a2387fc0998] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:09:58.520 INFO 11 --- [pool-scanner--15975222-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:09:58.743 INFO 11 --- [pool-ocp-schedules-12,,b759f53f08d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:09:59.799 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e24c3fd6be564d0e,d9b898aecf6f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=e24c3fd6be564d0e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:09:59.801 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e24c3fd6be564d0e,d9b898aecf6f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e24c3fd6be564d0e, duration=3 ms] 2023-05-15 14:09:59.907 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:09:59.908 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:10:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:10:00.001 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:10:00.001 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131000 2023-05-15 14:10:00.050 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:10:00.536 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.536 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=12, name=Refresh agent config, templateName=RefreshAgentConfigSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Refresh agent config, nodeCount=1, taskCount=1 2023-05-15 14:10:00.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.540 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Refresh agent config, taskDefinitionId=12, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.548 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:08:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.548 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:08:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:10:00.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.556 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.562 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.563 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=15, name=Sync system event, templateName=SyncSystemEventSchedule, scheduleRule=0 */10 * * * *, lastRunTime=2023-05-15 14:00:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.569 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync system event, nodeCount=2, taskCount=3 2023-05-15 14:10:00.569 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.569 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync system event, taskDefinitionId=15, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.580 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.580 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-05-15 14:10:00.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.588 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.588 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.620 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:10:00.620 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.620 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.623 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131954, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:00.625 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131954, name=Wait node, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=0, startTime=2023-05-15T14:10:00.621+08:00, endTime=null} 2023-05-15 14:10:00.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:09:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:10:00.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.629 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.635 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.635 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:09:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.637 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131952, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:00.638 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131952, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:10:00.634+08:00, endTime=null} 2023-05-15 14:10:00.648 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:10:00.648 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.648 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.651 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,87952e93d6f6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131954, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116345, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:00.620+08:00, sub_task_instance_id=131954}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:00.652 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131953, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:00.653 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131953, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:10:00.649+08:00, endTime=null} 2023-05-15 14:10:00.656 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,87952e93d6f6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131954, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:00.659 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.659 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:09:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.661 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,5083e638747d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131952, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116344, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:00.634+08:00, sub_task_instance_id=131952}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:00.663 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131957, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:00.664 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131957, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-05-15T14:10:00.661+08:00, endTime=null} 2023-05-15 14:10:00.667 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:10:00.667 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.667 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.672 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,0c7f8621b229] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131953, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116344, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:00.649+08:00, sub_task_instance_id=131953}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:00.673 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:31.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.673 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:09:31.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.674 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131951, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:00.676 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131951, name=Refresh all agent config, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.RefreshAllAgentConfigTask, seriesId=0, startTime=2023-05-15T14:10:00.671+08:00, endTime=null} 2023-05-15 14:10:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:10:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.679 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,5083e638747d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131952, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:00.681 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.681 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.683 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fed3c2682a19] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131957, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116346, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:00.661+08:00, sub_task_instance_id=131957}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:00.685 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-05-15 14:10:00.685 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.685 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.685 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,0c7f8621b229] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131953, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:00.685 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fed3c2682a19] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-05-15 14:10:00.691 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:05:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.691 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:05:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.692 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fed3c2682a19] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131957, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:00.692 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131951, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116343, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:00.671+08:00, sub_task_instance_id=131951}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:00.698 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2023-05-15 14:10:00.698 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2023-05-15 14:10:00.698 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2023-05-15 14:10:00.698 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2023-05-15 14:10:00.698 INFO 11 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2023-05-15 14:10:00.701 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:00.701 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:00.701 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.701 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:00.706 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.706 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.706 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.707 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:00.712 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:00.712 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.712 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:10:00.713 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.713 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.713 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:10:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:10:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:01.0, nextRun=2023-05-15 14:10:00.0, currentDate=Mon May 15 14:10:00 CST 2023 2023-05-15 14:10:00.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:09:01.0, nextRunTime=2023-05-15 14:10:00.0} 2023-05-15 14:10:00.734 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f214dea37b6b47d1,0e7c88ef32a6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=f214dea37b6b47d1, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=ocpcluster,1,false,2,] 2023-05-15 14:10:00.735 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f214dea37b6b47d1,0e7c88ef32a6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=f214dea37b6b47d1, duration=2 ms] 2023-05-15 14:10:00.736 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:00.736 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.737 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:10:00.737 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:00.737 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:00.738 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:10:00.767 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:00.767 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.767 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,8a1888fd8bb54825,bf5f497b0396] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=8a1888fd8bb54825, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-05-15 14:10:00.767 INFO 11 --- [ForkJoinPool-66062-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.10, 2982] 2023-05-15 14:10:00.769 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,8a1888fd8bb54825,bf5f497b0396] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=8a1888fd8bb54825, duration=2 ms] 2023-05-15 14:10:00.770 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType election_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.xxx.xx.14 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:10:00.771 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:00.779 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:10:00.779 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:10:00.779 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.xxx.xx.10 2023-05-15 14:10:00.779 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2023-05-15 14:10:00.790 INFO 11 --- [ForkJoinPool.commonPool-worker-7,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2023-05-15 14:10:00.791 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2023-05-15 14:10:00.791 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:00.795 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2023-05-15 14:10:00.797 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2023-05-15 14:10:00.799 INFO 11 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:00.803 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:00.805 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.805 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.805 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(agent.http.basic.auth.metricAuthEnabled=true), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}])]), params:null 2023-05-15 14:10:00.805 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:00.805 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.11, 2982] 2023-05-15 14:10:00.807 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.807 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.808 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.13, 2982] 2023-05-15 14:10:00.810 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:00.810 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:00.810 INFO 11 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2023-05-15 14:10:00.810 INFO 11 --- [ForkJoinPool-66062-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.12, 2982] 2023-05-15 14:10:00.824 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.10), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=ocpcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=1), (monagent.ob.zone.name=zone1), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/home/admin/ob), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=inactive), (monagent.pipeline.plan.monitor.status=inactive), (monagent.pipeline.sql.plan.status=inactive), (monagent.pipeline.transaction.status=inactive), (monagent.pipeline.session.status=inactive), (monagent.pipeline.slow.sql.status=inactive)]), params:null 2023-05-15 14:10:00.825 INFO 11 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone3), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.13), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:10:00.829 INFO 11 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2023-05-15 14:10:00.831 INFO 11 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2023-05-15 14:10:00.836 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone2), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.12), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:10:00.836 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.ob.monitor.password=xxx (monagent.host.ip=10.xxx.xx.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.ob.cluster.name=obcluster), (monagent.ob.cluster.id=1), (monagent.cluster.id=2), (monagent.ob.zone.name=zone1), (monagent.ob.sql.port=2981), (monagent.ob.rpc.port=2982), (ob.log.path=/data/oceanbase/redolog/ob), (ob.data.path=/data/oceanbase/storage/ob), (monagent.pipeline.ob.status=active), (ob.logcleaner.enabled=true), (ob.logtailer.enabled=true), (ob.install.path=/data/oceanbase/product/ob/oceanbase), (monagent.log.level=info), (mgragent.log.level=info), (monagent.monitordb.host=10.xxx.xx.10), (monagent.monitordb.port=2983), (monagent.monitordb.user=monitor_user@monitor_tenant#ocpcluster), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_db), (monagent.obproxy.cluster.id=3), (monagent.obproxy.cluster.name=obproxy), (monagent.host.ip=10.xxx.xx.11), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.log.level=info), (mgragent.log.level=info), (monagent.obproxy.sql.port=2983), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2984), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["is_out_of_memstore_mem=true","right_to_die_or_duty_to_live","on_fatal_error"],"level":1,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["partition table update task cost too much time to execute","clog disk is almost full","set_disk_error:attention!!!","error occurs on data disk or slog disk"],"level":2,"logLevels":["ERROR","WARNING"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"election_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"election_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"observer_all_error_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"obproxy_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"obproxy_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_execute_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"sql_execute_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme"],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"rootservice_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"expression":{"metric":"rootservice_log"},"expressionType":"Log","includeKeywords":[],"level":2,"logLevels":["ERROR"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_sql_query_slow","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"slow_sql_query_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.sql.audit.status=active), (monagent.pipeline.sql.plan.status=active), (monagent.pipeline.plan.monitor.status=active), (monagent.pipeline.transaction.status=active), (monagent.pipeline.session.status=active), (monagent.pipeline.slow.sql.status=active)]), params:null 2023-05-15 14:10:00.842 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2023-05-15 14:10:00.842 INFO 11 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2023-05-15 14:10:00.850 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ec07e544e6c4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131951, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.382 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116343, status:SUCCESS 2023-05-15 14:10:01.383 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116343, status:SUCCESS 2023-05-15 14:10:01.392 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116344, status:SUCCESS 2023-05-15 14:10:01.393 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116344, status:SUCCESS 2023-05-15 14:10:01.409 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116346, status:SUCCESS 2023-05-15 14:10:01.410 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116346, status:SUCCESS 2023-05-15 14:10:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,b59060e6a87d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131060, interval=60 2023-05-15 14:10:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,969e27e59105] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131060, interval=60 2023-05-15 14:10:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,b59060e6a87d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:10:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,b59060e6a87d] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131001, count=8, elapsed=1682446870519 2023-05-15 14:10:01.520 INFO 11 --- [pool-prometheus-minute-schedule-1,,969e27e59105] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:10:01.520 INFO 11 --- [pool-prometheus-minute-schedule-1,,969e27e59105] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131001, count=6, elapsed=1682446870519 2023-05-15 14:10:01.703 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131955, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.706 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131955, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=1, startTime=2023-05-15T14:10:01.701+08:00, endTime=null} 2023-05-15 14:10:01.716 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131958, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.718 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131958, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-05-15T14:10:01.713+08:00, endTime=null} 2023-05-15 14:10:01.728 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131959, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.731 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131959, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:10:01.726+08:00, endTime=null} 2023-05-15 14:10:01.732 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131955, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116345, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.701+08:00, sub_task_instance_id=131955}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.746 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131956, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.746 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57fe3ef4131b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131958, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116347, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.713+08:00, sub_task_instance_id=131958}, listMap={cluster_ids=[[1,1,2,1]], tenant_ids=[[2,3,6,4]]}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.746 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57fe3ef4131b] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [2, 3, 6, 4] 2023-05-15 14:10:01.749 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131956, name=Sync system event, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask, seriesId=2, startTime=2023-05-15T14:10:01.743+08:00, endTime=null} 2023-05-15 14:10:01.756 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,78c6d79f7ca8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131959, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116348, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.725+08:00, sub_task_instance_id=131959}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.759 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131962, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.759 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.759 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.759 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.761 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131962, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:10:01.756+08:00, endTime=null} 2023-05-15 14:10:01.763 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:10:01.764 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:10:01.764 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:10:01.764 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:10:01.764 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:10:01.767 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.767 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.767 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-05-15 08:00:12.528118, 2023-05-15 14:10:01.739, 1000] 2023-05-15 14:10:01.768 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2023-05-15 08:00:12.528118, endTime 2023-05-15 14:10:01.739 2023-05-15 14:10:01.769 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:01.769 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:01.770 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:01.770 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:01.770 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,78c6d79f7ca8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131959, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.779 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:01.783 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,cfd90bf9236f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131955, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131956, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116345, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.743+08:00, sub_task_instance_id=131956}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.791 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131963, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.793 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,289b0e9cdd02] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131962, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116351, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.756+08:00, sub_task_instance_id=131962}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.794 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,289b0e9cdd02] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:10:01.795 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131963, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-05-15T14:10:01.777+08:00, endTime=null} 2023-05-15 14:10:01.800 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:10:01.801 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:10:01.803 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:10:01.810 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:10:01.814 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:10:01.817 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131960, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.818 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.819 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.819 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.819 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131960, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:10:01.813+08:00, endTime=null} 2023-05-15 14:10:01.820 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:10:01.822 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:10:01.825 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.825 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.825 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,23e9cfa7892f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131963, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116352, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.776+08:00, sub_task_instance_id=131963}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.825 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,23e9cfa7892f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-05-15 14:10:01.826 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit, args: [2023-05-15 13:50:31.599358, 2023-05-15 14:10:01.79, 1000] 2023-05-15 14:10:01.829 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131961, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.831 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131961, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:10:01.827+08:00, endTime=null} 2023-05-15 14:10:01.832 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.832 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.833 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:10:01.834 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.835 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.835 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.836 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:10:01.837 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:10:01.837 INFO 11 --- [pool-3246256-thread-4,fb27d19334114ed7,d09ed2eb686f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:10:01.838 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.838 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:10:01.838 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.838 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.838 INFO 11 --- [pool-3246256-thread-2,fb27d19334114ed7,90262fa94353] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:10:01.839 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.839 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.839 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.840 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.840 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.840 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.841 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.842 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:10:01.843 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,a5a5292c4057] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131960, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116349, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.813+08:00, sub_task_instance_id=131960}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.843 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:10:01.843 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131964, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.843 INFO 11 --- [pool-3246256-thread-3,fb27d19334114ed7,a502489af085] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:10:01.844 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:10:01.845 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.845 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.845 INFO 11 --- [pool-3246256-thread-1,fb27d19334114ed7,0848d9c38d09] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:10:01.845 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131964, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-05-15T14:10:01.841+08:00, endTime=null} 2023-05-15 14:10:01.845 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:10:01.850 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.850 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.851 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:10:01.851 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:10:01.851 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:10:01.851 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:10:01.851 INFO 11 --- [pool-3246256-thread-5,fb27d19334114ed7,8f7d14310968] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:10:01.852 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,13f0bf9e0302] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131961, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116350, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.826+08:00, sub_task_instance_id=131961}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.855 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131965, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:01.856 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:10:01.857 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131965, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:10:01.852+08:00, endTime=null} 2023-05-15 14:10:01.857 INFO 11 --- [pool-3246256-thread-6,fb27d19334114ed7,ed38b1947f78] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:10:01.858 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,289b0e9cdd02] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:10:01.862 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131964, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116353, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.841+08:00, sub_task_instance_id=131964}, listMap={cluster_ids=[[1,1,2,1]], tenant_ids=[[2,4,6,3]]}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.862 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[2, 4, 6, 3] 2023-05-15 14:10:01.863 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.863 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.863 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.863 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.863 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,a5a5292c4057] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:10:01.864 INFO 11 --- [pool-subtask-executor28,fb27d19334114ed7,289b0e9cdd02] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131962, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.864 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,a5a5292c4057] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:10:01.865 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1684131001862000 2023-05-15 14:10:01.865 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1684130701703000 2023-05-15 14:10:01.870 INFO 11 --- [pool-subtask-executor31,fb27d19334114ed7,a5a5292c4057] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131960, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.872 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,13f0bf9e0302] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:10:01.872 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,13f0bf9e0302] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:10:01.874 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2totalLength:0 2023-05-15 14:10:01.874 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,13f0bf9e0302] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:10:01.874 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:01.874 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.874 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.875 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684130701000000, 1684131001000000] 2023-05-15 14:10:01.875 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.875 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.875 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,770b063922e9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131965, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116354, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:01.852+08:00, sub_task_instance_id=131965}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:01.875 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.878 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,13f0bf9e0302] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131961, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.883 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1684131001876000 2023-05-15 14:10:01.883 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1684130701686000 2023-05-15 14:10:01.884 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.885 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.885 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684130701000000, 1684131001000000] 2023-05-15 14:10:01.887 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 52 system events to database for cluster 2 2023-05-15 14:10:01.887 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 52 system events for cluster 2, startTime 2023-05-15 13:50:31.599358, endTime 2023-05-15 14:10:01.79, limit 1000, offset 0 2023-05-15 14:10:01.890 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,770b063922e9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131965, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.892 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2023-05-15 14:10:01.897 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1684131001894000 2023-05-15 14:10:01.897 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1684130701643000 2023-05-15 14:10:01.901 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:01.902 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.902 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.902 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.903 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:01.907 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:01.908 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:01.908 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND timestamp > :startTime AND timestamp <= :endTime ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2023-05-15 13:50:31.599358, 2023-05-15 14:10:01.79, 1000, 1000] 2023-05-15 14:10:01.910 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2023-05-15 13:50:31.599358, endTime 2023-05-15 14:10:01.79 2023-05-15 14:10:01.911 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2023-05-15 14:10:01.912 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,57fe3ef4131b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131958, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.913 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8f20d0a95158] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131956, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:01.914 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3currentTimeUs:1684131001912000 2023-05-15 14:10:01.915 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3diagnosisStartTimeUs:1684130701716000 2023-05-15 14:10:01.922 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-05-15 14:10:01.928 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,960f39a7e70a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131964, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:02.090 INFO 11 --- [pool-3246257-thread-2,fb27d19334114ed7,9e84fee79a12] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:10:02.157 INFO 11 --- [pool-3246257-thread-1,fb27d19334114ed7,67e3f9e2aabb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:10:02.159 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,23e9cfa7892f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-05-15 14:10:02.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,23e9cfa7892f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131963, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:02.494 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116345, status:SUCCESS 2023-05-15 14:10:02.495 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116345, status:SUCCESS 2023-05-15 14:10:02.505 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116347, status:SUCCESS 2023-05-15 14:10:02.507 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116347, status:SUCCESS 2023-05-15 14:10:02.517 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116348, status:SUCCESS 2023-05-15 14:10:02.519 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116348, status:SUCCESS 2023-05-15 14:10:02.530 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116349, status:SUCCESS 2023-05-15 14:10:02.532 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116349, status:SUCCESS 2023-05-15 14:10:02.542 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116350, status:SUCCESS 2023-05-15 14:10:02.544 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116350, status:SUCCESS 2023-05-15 14:10:02.554 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116351, status:SUCCESS 2023-05-15 14:10:02.555 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116351, status:SUCCESS 2023-05-15 14:10:02.561 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116352, status:SUCCESS 2023-05-15 14:10:02.562 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116352, status:SUCCESS 2023-05-15 14:10:02.569 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116353, status:SUCCESS 2023-05-15 14:10:02.570 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116353, status:SUCCESS 2023-05-15 14:10:02.577 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116354, status:SUCCESS 2023-05-15 14:10:02.578 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116354, status:SUCCESS 2023-05-15 14:10:02.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,9cabd6cd4b2f45f8,153895a46849] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=9cabd6cd4b2f45f8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:02.707 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,9cabd6cd4b2f45f8,153895a46849] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=9cabd6cd4b2f45f8, duration=6 ms] 2023-05-15 14:10:02.908 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:02.909 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:04.714 INFO 11 --- [pool-ocp-schedules-9,,6a8450735780] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:10:04.725 INFO 11 --- [pool-ocp-schedules-9,,6a8450735780] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:10:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:04.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:04.826917+08:00, lastProcessTime=2023-05-15T14:09:54.833468+08:00 2023-05-15 14:10:04.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:00:04.0, nextRun=2023-05-15 14:10:04.0, currentDate=Mon May 15 14:10:04 CST 2023 2023-05-15 14:10:04.891 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2023-05-15 14:00:04.0, nextRunTime=2023-05-15 14:10:04.0} 2023-05-15 14:10:04.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2023-05-15 14:10:04.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:04.896 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Partition rollover daemon, taskDefinitionId=1, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:04.896 INFO 11 --- [alarm-detect-8,,07957e342aea] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130984 ~ 1684130994 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.039, values={1684130990=246.039}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:04.918 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:04.826917+08:00 2023-05-15 14:10:05.909 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:05.910 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:05.922 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131966, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:05.924 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131966, name=Partition rollover daemon, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=2023-05-15T14:10:05.918+08:00, endTime=null} 2023-05-15 14:10:05.924 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:06.0, nextRun=2023-05-15 14:10:05.0, currentDate=Mon May 15 14:10:05 CST 2023 2023-05-15 14:10:05.924 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:09:06.0, nextRunTime=2023-05-15 14:10:05.0} 2023-05-15 14:10:05.927 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:10:05.928 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:05.928 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:05.944 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131966, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116355, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:05.918+08:00, sub_task_instance_id=131966}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:05.948 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2023-05-15 14:10:05.949 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2023-05-15 14:10:05.949 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.949 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.950 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:05.951 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.951 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.951 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2023-05-15 14:10:05.974 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.974 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.974 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:05.975 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.975 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.975 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2023-05-15 14:10:05.988 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.989 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.989 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:05.990 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:05.990 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:05.990 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2023-05-15 14:10:06.013 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.013 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.014 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.014 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.014 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.015 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2023-05-15 14:10:06.029 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.029 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.029 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.030 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.030 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.030 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2023-05-15 14:10:06.036 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.036 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.036 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.036 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.037 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.037 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2023-05-15 14:10:06.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2023-05-15 14:10:06.051 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.051 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.052 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.053 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.053 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.053 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2023-05-15 14:10:06.061 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.061 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.062 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.063 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.063 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.063 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2023-05-15 14:10:06.069 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.069 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.070 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.071 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2023-05-15 14:10:06.081 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.081 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.081 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.082 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.082 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.082 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2023-05-15 14:10:06.097 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.097 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.098 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.098 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.098 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.099 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2023-05-15 14:10:06.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2023-05-15 14:10:06.107 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:06.114 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.114 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.115 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.115 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.115 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.116 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2023-05-15 14:10:06.120 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:06.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.122 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:06.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.123 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2023-05-15 14:10:06.124 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:06.128 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.128 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.129 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.129 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.129 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.130 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2023-05-15 14:10:06.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.136 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2023-05-15 14:10:06.141 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.141 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.141 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.142 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.142 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.145 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2023-05-15 14:10:06.153 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.153 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.153 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2023-05-15 14:10:06.160 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.160 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.161 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.161 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.161 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.162 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2023-05-15 14:10:06.167 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.167 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.168 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2023-05-15 14:10:06.174 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.174 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.174 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.175 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.175 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.176 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2023-05-15 14:10:06.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.182 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.183 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2023-05-15 14:10:06.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.187 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.188 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.188 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.188 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.189 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2023-05-15 14:10:06.193 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.193 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.194 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.194 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.195 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.195 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2023-05-15 14:10:06.200 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.200 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.200 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.201 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.201 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.201 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2023-05-15 14:10:06.206 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.206 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.206 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.207 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.207 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.207 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2023-05-15 14:10:06.211 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.211 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.212 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.212 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.212 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.213 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2023-05-15 14:10:06.218 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.218 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.218 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.218 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.218 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.219 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2023-05-15 14:10:06.223 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.224 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.224 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.225 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2023-05-15 14:10:06.230 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.230 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.232 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2023-05-15 14:10:06.236 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.236 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.237 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.237 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.237 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.238 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2023-05-15 14:10:06.242 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.242 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.242 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.243 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.243 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.243 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2023-05-15 14:10:06.251 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.251 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.252 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.252 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.253 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat 2023-05-15 14:10:06.260 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.260 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.261 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.261 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.261 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.262 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2023-05-15 14:10:06.267 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.267 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.267 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.268 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2023-05-15 14:10:06.272 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.272 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.273 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2023-05-15 14:10:06.277 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.277 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.277 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.278 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.278 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.278 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2023-05-15 14:10:06.281 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.281 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2023-05-15 14:10:06.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.286 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.287 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.287 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.287 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2023-05-15 14:10:06.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.298 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.298 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.298 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2023-05-15 14:10:06.317 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.317 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.317 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.318 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.318 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.319 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2023-05-15 14:10:06.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.327 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.327 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.328 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2023-05-15 14:10:06.336 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.336 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.336 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.337 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.337 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.338 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2023-05-15 14:10:06.342 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.342 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.342 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.343 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.343 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.344 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2023-05-15 14:10:06.349 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.349 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.350 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.350 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.350 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.351 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2023-05-15 14:10:06.355 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2023-05-15 14:10:06.355 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2023-05-15 14:10:06.356 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.356 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.357 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.358 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2023-05-15 14:10:06.365 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.365 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.365 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:10:06.366 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:10:06.366 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:06.367 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2023-05-15 14:10:06.372 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2023-05-15 14:10:06.380 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8544730c3607] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131966, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:06.665 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116355, status:SUCCESS 2023-05-15 14:10:06.666 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116355, status:SUCCESS 2023-05-15 14:10:06.952 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131967, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:06.974 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131967, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:10:06.948+08:00, endTime=null} 2023-05-15 14:10:07.005 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,05bad07daa40] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131967, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116356, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:06.948+08:00, sub_task_instance_id=131967}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:07.025 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,c71587a6ccb1] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:10:07.026 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,0d4357490a56] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:10:07.026 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,89c74ff7178f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:10:07.064 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:07.064 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,c71587a6ccb1] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:10:07.065 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,89c74ff7178f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:10:07.065 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,0d4357490a56] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:10:07.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,05bad07daa40] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131967, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:07.699 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116356, status:SUCCESS 2023-05-15 14:10:07.700 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116356, status:SUCCESS 2023-05-15 14:10:08.269 INFO 11 --- [pool-overview-container-3,,88293caa2def] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:08.516 INFO 11 --- [pool-scanner--15975315-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:10:08.764 INFO 11 --- [pool-ocp-schedules-15,,19841b137a35] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:08.910 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:08.911 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:11.912 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:11.913 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:11.928 INFO 11 --- [pool-ocp-schedules-6,,6fda2b1fd68c] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:11.928 INFO 11 --- [pool-ocp-schedules-6,,6fda2b1fd68c] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:12.442 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:13.954 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,0fdb8a33473f48ff,52117329059f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=0fdb8a33473f48ff, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:13.958 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,0fdb8a33473f48ff,52117329059f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=0fdb8a33473f48ff, duration=4 ms] 2023-05-15 14:10:14.571 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130940, nodeCount=1000, affectRows=1000 2023-05-15 14:10:14.572 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130940, nodeCount=776, affectRows=776 2023-05-15 14:10:14.577 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130940, nodeCount=776, validCount=776 2023-05-15 14:10:14.578 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684130940, nodeCount=914, affectRows=914 2023-05-15 14:10:14.579 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130940, nodeCount=1000, validCount=1000 2023-05-15 14:10:14.583 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684130940, nodeCount=914, validCount=914 2023-05-15 14:10:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:14.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:14.830953+08:00, lastProcessTime=2023-05-15T14:10:04.826917+08:00 2023-05-15 14:10:14.856 INFO 11 --- [alarm-detect-13,,be485d49ebd9] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684130994 ~ 1684131004 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.084, values={1684131000=246.084}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:14.891 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:14.830953+08:00 2023-05-15 14:10:14.913 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:14.914 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:16.110 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:16.122 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:16.124 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:16.126 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:16.978 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,2ef7ba9a41fa42d7,2989152a3f81] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=2ef7ba9a41fa42d7, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:16.983 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,2ef7ba9a41fa42d7,2989152a3f81] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=2ef7ba9a41fa42d7, duration=5 ms] 2023-05-15 14:10:17.067 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:17.914 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:17.915 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:18.268 INFO 11 --- [pool-overview-container-4,,451afa9927c8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:18.518 INFO 11 --- [pool-scanner--15975415-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:18.782 INFO 11 --- [pool-ocp-schedules-12,,8ed4bb34c6db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:19.773 INFO 11 --- [pool-ocp-schedules-10,,baff8970d201] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:10:19.773 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:10:19.773 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:10:19.773 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:10:19.774 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:10:19.774 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:10:19.780 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:19.780 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:19.781 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:19.781 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:19.781 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:19.807 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:10:19.807 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:10:19.809 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:10:19.809 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:10:19.809086722+08:00 2023-05-15 14:10:19.809 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:10:19.809 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:10:19.809+08:00 2023-05-15 14:10:19.810 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:10:19.809665492+08:00 2023-05-15 14:10:19.810 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:10:19.810+08:00 2023-05-15 14:10:19.811 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:10:19.811300953+08:00 2023-05-15 14:10:19.811 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:10:19.811+08:00 2023-05-15 14:10:19.811 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:10:19.811560755+08:00 2023-05-15 14:10:19.812 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:10:19.811+08:00 2023-05-15 14:10:19.813 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:10:19.810+08:00 and status AVAILABLE 2023-05-15 14:10:19.813 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:10:19.813 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:10:19.809+08:00 and status AVAILABLE 2023-05-15 14:10:19.814 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:10:19.811+08:00 and status AVAILABLE 2023-05-15 14:10:19.814 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:10:19.811+08:00 and status AVAILABLE 2023-05-15 14:10:19.815 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:10:19.814891873+08:00 2023-05-15 14:10:19.815 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:10:19.815+08:00 2023-05-15 14:10:19.818 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:10:19.815+08:00 and status AVAILABLE 2023-05-15 14:10:19.818 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:10:19.818 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:10:19.819 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:10:19.820 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:10:19.829 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131019 0 0 , error= 2023-05-15 14:10:19.832 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131019 0 0 , error= 2023-05-15 14:10:19.832 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:10:19.838 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:10:19.832+08:00 2023-05-15 14:10:19.836 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131019 0 0 , error= 2023-05-15 14:10:19.838 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:10:19.838+08:00 2023-05-15 14:10:19.841 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131019 0 0 , error= 2023-05-15 14:10:19.843 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:10:19.838+08:00 2023-05-15 14:10:19.856 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131019 0 0 , error= 2023-05-15 14:10:19.857 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:10:19.843+08:00 2023-05-15 14:10:19.857 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:10:19.857+08:00 2023-05-15 14:10:19.860 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:10:19.832+08:00 and status AVAILABLE 2023-05-15 14:10:19.860 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:10:19.838+08:00 and status AVAILABLE 2023-05-15 14:10:19.861 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:10:19.838+08:00 and status AVAILABLE 2023-05-15 14:10:19.862 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:10:19.843+08:00 and status AVAILABLE 2023-05-15 14:10:19.862 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:10:19.857+08:00 and status AVAILABLE 2023-05-15 14:10:19.863 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:10:19.863 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:10:19.864 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:10:19.864 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:10:19.865 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:10:19.874 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:19.874 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:19.876 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:19.876 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:19.876 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:19.896 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:10:19.896 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:10:19.900 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:10:19.901 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:10:19.902 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:10:19.944 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:10:19.950 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:10:19.957 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:10:19.958 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:10:19.960 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:10:19.961 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:10:19.968 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:10:19.968 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:10:19.973 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:10:19.973 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:10:19.980 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:10:19.982 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:10:19.988 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:10:19.998 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:10:20.017 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:10:20.023 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:10:20.026 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:10:20.035 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:10:20.036 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:10:20.040 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:10:20.041 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:10:20.046 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:10:20.053 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:10:20.056 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:10:20.059 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:10:20.062 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:10:20.074 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:10:20.097 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:10:20.102 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:10:20.113 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:10:20.114 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:10:20.116 INFO 11 --- [pool-Check host status13,,da3422c661d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:10:20.118 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:10:20.128 INFO 11 --- [pool-Check host status30,,90ad685e8265] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:10:20.131 INFO 11 --- [pool-Check host status14,,15eee5b2cde4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:10:20.131 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:10:20.133 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:10:20.141 INFO 11 --- [pool-Check host status8,,cd52f9c955b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:10:20.220 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:10:20.233 INFO 11 --- [pool-Check host status1,,54c98942ab0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:10:20.254 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:10:20.254 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:10:20.254 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:10:20.254 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:10:20.254 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:10:20.261 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:20.261 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:20.261 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:20.261 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:20.262 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:20.285 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:10:20.287 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:10:20.287 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:10:20.288 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:10:20.295 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:10:20.304 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:10:20.304 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:10:20.304 INFO 11 --- [pool-Check host status17,,e8d0ae2a39d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:10:20.306 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:10:20.306 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:10:20.306 INFO 11 --- [pool-Check host status2,,ecf41867cf54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:10:20.307 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:10:20.308 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:10:20.308 INFO 11 --- [pool-Check host status19,,bfba09278493] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:10:20.309 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:10:20.309 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:10:20.309 INFO 11 --- [pool-Check host status28,,de0108e2dbc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:10:20.359 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:10:20.359 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:10:20.360 INFO 11 --- [pool-Check host status3,,fa800b57e624] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:10:20.375 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:20.0, nextRun=2023-05-15 14:10:20.0, currentDate=Mon May 15 14:10:20 CST 2023 2023-05-15 14:10:20.376 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:09:20.0, nextRunTime=2023-05-15 14:10:20.0} 2023-05-15 14:10:20.380 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:10:20.380 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:20.380 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:20.915 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:20.917 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:21.172 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131968, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:21.173 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131968, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:10:21.170+08:00, endTime=null} 2023-05-15 14:10:21.193 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2e45f4c0fa9a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131968, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116357, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:21.170+08:00, sub_task_instance_id=131968}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:21.195 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2e45f4c0fa9a] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:10:21.195 INFO 11 --- [ForkJoinPool-66065-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:10:21.195 INFO 11 --- [ForkJoinPool-66065-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:10:21.201 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2e45f4c0fa9a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131968, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:21.837 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:21.841 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:21.846 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:21.889 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-05-15 14:10:21.916 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:10:21.932 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:10:21.948 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:10:21.970 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116357, status:SUCCESS 2023-05-15 14:10:21.971 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116357, status:SUCCESS 2023-05-15 14:10:21.971 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:10:21.993 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:10:22.009 INFO 11 --- [pool-ocp-schedules-18,,0acbcb3b51bf] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=5 2023-05-15 14:10:22.358 INFO 11 --- [pool-scanner--15975416-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:22.379 INFO 11 --- [pool-scanner--15975417-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:22.402 INFO 11 --- [pool-scanner--15975418-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:22.424 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:22.429 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:22.472 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:22.476 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:22.492 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:22.495 INFO 11 --- [pool-ocp-schedules-10,,5ef8003618d8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:23.918 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:23.919 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:24.825 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:24.824872+08:00, lastProcessTime=2023-05-15T14:10:14.830953+08:00 2023-05-15 14:10:24.863 INFO 11 --- [alarm-detect-6,,fc7a553072f6] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131004 ~ 1684131014 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.084, values={1684131010=246.084}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:24.864 INFO 11 --- [alarm-detect-4,,8f9bb4dfbbf3] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130954 ~ 1684131014 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389397.988, values={1684131000=389397.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:10:24.865 INFO 11 --- [alarm-detect-4,,8f9bb4dfbbf3] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684130954 ~ 1684131014 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994195.358, values={1684131000=994195.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:24.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:24.824872+08:00 2023-05-15 14:10:25.528 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:25.0, nextRun=2023-05-15 14:10:25.0, currentDate=Mon May 15 14:10:25 CST 2023 2023-05-15 14:10:25.528 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:09:25.0, nextRunTime=2023-05-15 14:10:25.0} 2023-05-15 14:10:25.538 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:10:25.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:25.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:26.112 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:26.124 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:26.126 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:26.128 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:26.257 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131969, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:26.259 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131969, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:10:26.255+08:00, endTime=null} 2023-05-15 14:10:26.282 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131969, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116358, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:26.255+08:00, sub_task_instance_id=131969}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:10:26.322 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:10:26.323 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:10:26.324 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:10:26.324 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:10:26.325 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:10:26.331 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,7d4a7f895ad7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131969, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:26.620 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,04e8f279fa574d39,e63a471becfe] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=04e8f279fa574d39, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:26.625 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,04e8f279fa574d39,e63a471becfe] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=04e8f279fa574d39, duration=6 ms] 2023-05-15 14:10:26.919 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:26.920 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:27.070 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:27.082 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116358, status:SUCCESS 2023-05-15 14:10:27.084 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116358, status:SUCCESS 2023-05-15 14:10:27.445 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:28.269 INFO 11 --- [pool-overview-container-8,,3ee5c87e7a98] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:28.518 INFO 11 --- [pool-scanner--15975516-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:28.801 INFO 11 --- [pool-ocp-schedules-15,,3deb2046a029] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:29.920 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:29.922 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:30.441 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.473 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.479 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.485 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.491 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.493 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.494 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.495 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.496 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.498 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.499 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.500 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.501 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.502 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.503 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.504 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.505 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.512 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.520 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.529 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.535 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.537 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.538 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.540 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.541 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:04:30.281, end=2023-05-15T14:09:30.42} 2023-05-15 14:10:30.541 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:10:30.702 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:09:31.0, nextRun=2023-05-15 14:10:30.0, currentDate=Mon May 15 14:10:30 CST 2023 2023-05-15 14:10:30.702 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:09:31.0, nextRunTime=2023-05-15 14:10:30.0} 2023-05-15 14:10:30.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:10:30.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:30.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:30.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:30.722 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:10:30.0, currentDate=Mon May 15 14:10:30 CST 2023 2023-05-15 14:10:30.722 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:10:30.0} 2023-05-15 14:10:30.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:10:30.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:10:30.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:10:31.342 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131970, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:31.344 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131970, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:10:31.339+08:00, endTime=null} 2023-05-15 14:10:31.358 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131971, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:31.360 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131971, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:10:31.355+08:00, endTime=null} 2023-05-15 14:10:31.366 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131970, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116359, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:31.339+08:00, sub_task_instance_id=131970}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:31.371 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131972, operation:EXECUTE, state: RUNNING 2023-05-15 14:10:31.373 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131972, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:10:31.368+08:00, endTime=null} 2023-05-15 14:10:31.380 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131971, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116359, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:31.355+08:00, sub_task_instance_id=131971}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:10:31.393 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8c28dd622284] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131972, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116360, task_operation=execute, latest_execution_start_time=2023-05-15T14:10:31.368+08:00, sub_task_instance_id=131972}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:10:31.393 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8c28dd622284] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:10:31.405 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.405 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.405 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.406 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.407 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.407 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.407 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:10:31.409 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:10:31.409 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:10:31.410 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:10:31.410 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:10:31.410 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:10:31.410 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:10:31.419 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.419 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.420 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.421 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:10:31.422 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.423 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.423 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.425 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:10:31.425 INFO 11 --- [pool-3246415-thread-4,fb27d19334114ed7,864210307bac] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:10:31.425 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.425 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.432 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:10:31.433 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.433 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:10:31.433 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.433 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.434 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.434 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.434 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.434 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.435 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:10:31.435 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:10:31.435 INFO 11 --- [pool-3246415-thread-3,fb27d19334114ed7,067d386b3c57] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:10:31.435 INFO 11 --- [pool-3246415-thread-1,fb27d19334114ed7,2efd59dbc43e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:10:31.436 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:10:31.436 INFO 11 --- [pool-3246415-thread-2,fb27d19334114ed7,6cf7158141bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:10:31.439 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.439 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.439 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.439 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.440 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.440 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.440 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.440 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:10:31.441 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:10:31.445 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.446 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:10:31.446 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:10:31.446 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.446 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.446 INFO 11 --- [pool-3246415-thread-5,fb27d19334114ed7,8185a42a2761] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:10:31.446 INFO 11 --- [pool-3246415-thread-6,fb27d19334114ed7,024e892c3882] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:10:31.447 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.447 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8c28dd622284] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:10:31.451 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.451 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.451 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.451 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.451 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.452 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:10:31.452 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,8c28dd622284] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131972, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:31.457 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.457 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.458 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.461 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.461 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.462 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.471 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:10:31.472 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.472 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.473 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.480 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.480 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.480 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:10:31.484 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.484 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.484 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.491 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.491 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.491 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:10:31.495 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.495 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.495 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.501 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.501 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.501 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:10:31.505 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:10:31.519 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:10:31.537 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.537 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.537 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.537 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.538 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.539 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.539 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.539 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.539 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.540 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.541 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.541 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.541 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.546 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.546 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.547 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.553 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.553 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.553 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.553 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.554 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.555 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.555 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.555 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.570 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.571 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.571 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.571 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.572 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.579 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.579 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.580 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.580 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.580 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.582 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.597 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.597 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.598 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.598 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.598 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.599 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.599 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.600 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.616 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.617 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.617 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.618 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.618 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.623 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.623 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.624 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.627 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.628 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.628 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.628 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.629 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.630 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.630 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.631 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.646 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.647 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.647 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.648 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.648 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.648 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.648 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.648 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.649 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.649 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.650 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.653 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.653 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.653 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.654 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:10:31.657 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:10:31.670 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.670 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.672 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.672 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.672 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:10:31.681 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.682 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.682 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.682 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.682 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.683 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.683 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.683 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:10:31.683 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:10:31.684 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.685 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.685 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:10:31.686 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.686 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.686 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.692 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.700 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:31.701 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.701 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.701 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.702 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:31.704 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.706 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.706 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.706 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.707 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:10:31.708 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:31.708 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.709 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:10:31.715 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.717 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:10:31.717 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.717 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.718 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:10:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:10:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:10:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:10:31.725 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,675c7759aa2c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131971, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:31.726 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:10:31.728 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.729 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.729 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:10:31.744 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.746 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:10:31.747 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:31.747 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:31.747 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:10:31.756 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:10:31.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:10:31.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:10:31.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:10:31.764 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b84709c363ee] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131970, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:10:31.933 INFO 11 --- [pool-ocp-schedules-18,,f5f0598c879e] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:31.933 INFO 11 --- [pool-ocp-schedules-18,,f5f0598c879e] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:32.057 INFO 11 --- [pool-ocp-schedules-9,,ac6d8785af38] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:10:32.060 INFO 11 --- [pool-ocp-schedules-9,,ac6d8785af38] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:10:32.068 INFO 11 --- [pool-ocp-schedules-11,,5f58e2ae6025] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:10:32.070 INFO 11 --- [pool-ocp-schedules-11,,5f58e2ae6025] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:10:32.084 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.085 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.085 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.085 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.086 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.086 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.087 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.088 INFO 11 --- [ForkJoinPool-66066-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:10:32.090 INFO 11 --- [pool-ocp-schedules-12,,160cc79c6517] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:10:32.091 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.091 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.091 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.092 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.092 INFO 11 --- [pool-ocp-schedules-12,,160cc79c6517] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:10:32.096 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.096 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.097 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.101 INFO 11 --- [ForkJoinPool-66066-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:10:32.122 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.123 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.123 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.123 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.125 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.125 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.125 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:10:32.135 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.136 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.136 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.137 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.141 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.141 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.142 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:10:32.176 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:32.177 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.177 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.177 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.178 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.179 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.179 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.179 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:10:32.181 WARN 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:10:32.181 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.181 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.182 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:32.183 INFO 11 --- [ForkJoinPool-66068-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:10:32.184 INFO 11 --- [ForkJoinPool-66068-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:10:32.184 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:10:32.194 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116359, status:SUCCESS 2023-05-15 14:10:32.196 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116359, status:SUCCESS 2023-05-15 14:10:32.196 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.197 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:10:32.207 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116360, status:SUCCESS 2023-05-15 14:10:32.208 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116360, status:SUCCESS 2023-05-15 14:10:32.210 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.220 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:32.221 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.221 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.222 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.223 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.224 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:10:32.224 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.224 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.227 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.227 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.228 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:10:32.232 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.232 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.232 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:32.234 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:10:32.234 INFO 11 --- [ForkJoinPool-66068-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:10:32.235 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:10:32.252 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.252 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:10:32.267 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.267 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:10:32.282 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.282 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:10:32.300 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.300 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:10:32.317 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.318 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:10:32.333 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:10:32.354 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:32.355 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.356 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.356 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.357 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.361 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:32.361 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:32.362 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:32.386 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:10:32.386 INFO 11 --- [ForkJoinPool-66068-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:10:32.922 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:32.923 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:33.831 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,bff98c0bd06343b4,de96474e36f7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=bff98c0bd06343b4, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:10:33.833 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,bff98c0bd06343b4,de96474e36f7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=bff98c0bd06343b4, duration=3 ms] 2023-05-15 14:10:34.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:34.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:34.833357+08:00, lastProcessTime=2023-05-15T14:10:24.824872+08:00 2023-05-15 14:10:34.886 INFO 11 --- [alarm-detect-14,,06c92fe3da38] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131014 ~ 1684131024 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131020=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:34.911 INFO 11 --- [alarm-detect-9,,cc653c45e8c5] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130964 ~ 1684131024 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389397.938, values={1684131000=389397.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:10:34.911 INFO 11 --- [alarm-detect-9,,cc653c45e8c5] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684130964 ~ 1684131024 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994195.107, values={1684131000=994195.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:34.912 INFO 11 --- [alarm-detect-9,,cc653c45e8c5] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:10:34.922 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:34.833357+08:00 2023-05-15 14:10:35.923 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:35.924 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:36.114 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:36.126 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:36.128 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:36.130 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:36.599 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,587251024add45f0,efc385b6dba2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=587251024add45f0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:36.604 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,587251024add45f0,efc385b6dba2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=587251024add45f0, duration=5 ms] 2023-05-15 14:10:37.074 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:38.270 INFO 11 --- [pool-overview-container-6,,622cf5eee3f4] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:10:38.270 INFO 11 --- [pool-overview-container-6,,622cf5eee3f4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:38.514 INFO 11 --- [pool-scanner--15975627-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:38.822 INFO 11 --- [pool-ocp-schedules-1,,036e34fcad2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:38.925 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:38.925 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:40.471 INFO 11 --- [pool-ocp-schedules-7,,2afe988c4bbd] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:10:41.926 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:41.928 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:42.214 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,730fbf397d8e46d9,11355945c317] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=730fbf397d8e46d9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:42.221 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,730fbf397d8e46d9,11355945c317] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=730fbf397d8e46d9, duration=7 ms] 2023-05-15 14:10:42.447 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:43.312 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,3841a59581d74e86,50f954d3bc21] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=3841a59581d74e86, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:43.318 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,3841a59581d74e86,50f954d3bc21] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=3841a59581d74e86, duration=6 ms] 2023-05-15 14:10:44.390 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,af456d79d66b4eb6,9d3afe338551] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch?id=2&tenantId=6, client=192.168.80.229, traceId=af456d79d66b4eb6, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitch(Long,Long), args=2,6,] 2023-05-15 14:10:44.391 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,93ca78326ca04bb5,cc835e68579d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events?null, client=192.168.80.229, traceId=93ca78326ca04bb5, method=PaginatedResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchEvents(Long,Long,OffsetDateTime,OffsetDateTime,Pageable), args=2,6,Page request [number: 0, size 2000, sort: timestamp: ASC],] 2023-05-15 14:10:44.393 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/performance/config?id=2&tenantId=6, client=192.168.80.229, traceId=8a78e95af3f240e8, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getClusterPerfConfig(Long,Long), args=2,6,] 2023-05-15 14:10:44.402 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,93ca78326ca04bb5,cc835e68579d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/events client=192.168.80.229, traceId=93ca78326ca04bb5, duration=13 ms] 2023-05-15 14:10:44.413 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,af456d79d66b4eb6,9d3afe338551] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/switch client=192.168.80.229, traceId=af456d79d66b4eb6, duration=24 ms] 2023-05-15 14:10:44.415 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:44.421 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.421 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.422 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() REGEXP 'OceanBase.*?CE' 2023-05-15 14:10:44.423 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.423 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.424 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2023-05-15 14:10:44.437 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/outlines?null, client=192.168.80.229, traceId=48fa624cd6f04504, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObOutlineController.getSqlOutline(Long,Long,String,String,OffsetDateTime,OffsetDateTime,Boolean), args=2,6,] 2023-05-15 14:10:44.462 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:10:44.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:10:44.475 ERROR 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1049956) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1049956) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:10:44.478 ERROR 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] com.alibaba.druid.pool.DruidDataSource : {dataSource-102} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1049956) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1049956) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:10:44.478 INFO 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] com.alibaba.druid.pool.DruidDataSource : {dataSource-102,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:10:44.478 INFO 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] com.alibaba.druid.pool.DruidDataSource : {dataSource-102} closing ... 2023-05-15 14:10:44.482 INFO 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] com.alibaba.druid.pool.DruidDataSource : {dataSource-102} closed 2023-05-15 14:10:44.482 ERROR 11 --- [obsdk-init-thread-3,48fa624cd6f04504,6e7c68b0438b] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1049956) Get Location Cache Fail 2023-05-15 14:10:44.482 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1049956) Get Location Cache Fail 2023-05-15 14:10:44.483 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1049956) Get Location Cache Fail 2023-05-15 14:10:44.483 WARN 11 --- [http-nio-0.0.0.0-8080-exec-52,48fa624cd6f04504,85abe2a40338] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/outlines client=192.168.80.229, traceId=48fa624cd6f04504, duration=47 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:10:44.491 ERROR 11 --- [Druid-ConnectionPool-Create-1489368917,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1049959) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1049959) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:10:44.545 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d4db2d0cdd314b71,edf99507d5ae] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents?startTime=2023-05-15T13%3A40%3A44%2B08%3A00&endTime=2023-05-15T14%3A10%3A44%2B08%3A00, client=192.168.80.229, traceId=d4db2d0cdd314b71, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchOffIntervalLimit3(Long,Long,OffsetDateTime,OffsetDateTime), args=2,6,2023-05-15T13:40:44+08:00,2023-05-15T14:10:44+08:00,] 2023-05-15 14:10:44.549 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/topSql?startTime=2023-05-15T13%3A40%3A44%2B08%3A00&endTime=2023-05-15T14%3A10%3A44%2B08%3A00&inner=false, client=192.168.80.229, traceId=81889e80dfb84d5b, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,6,2023-05-15T13:40:44+08:00,2023-05-15T14:10:44+08:00,false,] 2023-05-15 14:10:44.549 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:10:44.550 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:10:44.561 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d4db2d0cdd314b71,edf99507d5ae] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql/status/switchOffEvents client=192.168.80.229, traceId=d4db2d0cdd314b71, duration=18 ms] 2023-05-15 14:10:44.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.573 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2023-05-15 14:10:44.575 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.576 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.576 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.576 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:44.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:44.586 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:40:44, end=2023-05-15T14:10:44}, clusterId=2, tenantId=6, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:10:44.589 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:40:44, end=2023-05-15T14:10:44}} 2023-05-15 14:10:44.638 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:10:44.646 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:44.647 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.647 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.647 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.648 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:44.653 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.653 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.654 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:44.656 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:44.656 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:44.657 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:10:44.672 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:10:44.673 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:10:44.681 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,81889e80dfb84d5b,2e5ecdf65964] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/topSql client=192.168.80.229, traceId=81889e80dfb84d5b, duration=134 ms] 2023-05-15 14:10:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:44.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:44.832356+08:00, lastProcessTime=2023-05-15T14:10:34.833357+08:00 2023-05-15 14:10:44.859 INFO 11 --- [alarm-detect-12,,b8f40bdca38d] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131024 ~ 1684131034 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131030=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:44.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,8a78e95af3f240e8,9f040622e4c9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/performance/config client=192.168.80.229, traceId=8a78e95af3f240e8, duration=494 ms] 2023-05-15 14:10:44.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:44.832356+08:00 2023-05-15 14:10:44.929 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:44.930 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:45.421 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/61E88A3E6ACB873A6CCB8295FD24DEF5/text?startTime=2023-05-15T13%3A40%3A44%2B08%3A00&endTime=2023-05-15T14%3A10%3A44%2B08%3A00, client=192.168.80.229, traceId=491b84c201f34247, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,61E88A3E6ACB873A6CCB8295FD24DEF5,2023-05-15T13:40:44+08:00,2023-05-15T14:10:44+08:00,] 2023-05-15 14:10:45.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.449 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.449 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:45.450 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:45.454 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.454 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:45.455 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:45.467 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:45.468 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.468 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.468 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:45.469 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:45.483 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:45.483 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:45.484 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:10:45.513 INFO 11 --- [pool-ocp-schedules-11,,64ecd23d1b04] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:10:45.597 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,491b84c201f34247,53d87dc1cac5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/61E88A3E6ACB873A6CCB8295FD24DEF5/text client=192.168.80.229, traceId=491b84c201f34247, duration=178 ms] 2023-05-15 14:10:46.067 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/67FB0008AEAD4B0DC76BF693C53E374B/text?startTime=2023-05-15T13%3A40%3A44%2B08%3A00&endTime=2023-05-15T14%3A10%3A44%2B08%3A00, client=192.168.80.229, traceId=90248fd695e84823, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,67FB0008AEAD4B0DC76BF693C53E374B,2023-05-15T13:40:44+08:00,2023-05-15T14:10:44+08:00,] 2023-05-15 14:10:46.099 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.100 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.100 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:46.100 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:46.105 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.105 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:46.106 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:10:46.116 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:46.119 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:10:46.120 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.120 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.120 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:46.121 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:10:46.125 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:10:46.125 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:10:46.126 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:10:46.128 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:46.130 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:46.132 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:46.181 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e9de4c1f1f0b4dc9,1d92980877e8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObIDCRegionInfo&User_ID=alibaba&UID=test&ObRegion=obcluster, client=10.xxx.xx.11, traceId=e9de4c1f1f0b4dc9, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObIDCRegionInfo(String,Long,String,Long,Integer), args=obcluster,0,] 2023-05-15 14:10:46.183 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,e9de4c1f1f0b4dc9,1d92980877e8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=e9de4c1f1f0b4dc9, duration=3 ms] 2023-05-15 14:10:46.234 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,90248fd695e84823,12ba8815b7d4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/67FB0008AEAD4B0DC76BF693C53E374B/text client=192.168.80.229, traceId=90248fd695e84823, duration=169 ms] 2023-05-15 14:10:47.063 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059} 2023-05-15 14:10:47.077 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:47.088 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.088 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.090 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.090 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.091 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.091 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.099 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.099 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.103 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.103 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.104 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:06:47.001, end=2023-05-15T14:08:47.059},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:10:47.104 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:10:47.105 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:10:47.930 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:47.931 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:48.269 INFO 11 --- [pool-overview-container-2,,bb9c0b230314] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:48.513 INFO 11 --- [pool-scanner--15975727-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:48.846 INFO 11 --- [pool-ocp-schedules-6,,1bcb74f672f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:50.876 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,de4a0d167d3d4658,6b13bbf6d9a0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=de4a0d167d3d4658, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:10:50.881 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,de4a0d167d3d4658,6b13bbf6d9a0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=de4a0d167d3d4658, duration=6 ms] 2023-05-15 14:10:50.931 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:50.932 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:51.841 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:51.846 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:51.856 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:51.938 INFO 11 --- [pool-ocp-schedules-9,,e8320cdabe68] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:51.938 INFO 11 --- [pool-ocp-schedules-9,,e8320cdabe68] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:10:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,9199102da582] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40060} 2023-05-15 14:10:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,9199102da582] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:10:53.932 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:53.934 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:10:54.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:10:54.829925+08:00, lastProcessTime=2023-05-15T14:10:44.832356+08:00 2023-05-15 14:10:54.868 INFO 11 --- [alarm-detect-12,,f667eaae789c] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131034 ~ 1684131044 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131040=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:10:54.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:10:54.829925+08:00 2023-05-15 14:10:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,f215e90493d8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:10:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,e5626bb8bae1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:10:56.119 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:56.130 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:56.132 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:56.134 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:56.934 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:56.935 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:10:57.080 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:57.450 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:10:58.270 INFO 11 --- [pool-overview-container-4,,017788135db1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:10:58.518 INFO 11 --- [pool-scanner--15975821-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:10:58.865 INFO 11 --- [pool-ocp-schedules-8,,b15c4572dd2a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:10:59.935 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:10:59.936 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:11:00.000 INFO 11 --- [pool-ocp-schedules-9,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:11:00.001 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:11:00.001 INFO 11 --- [pool-ocp-schedules-9,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131060 2023-05-15 14:11:00.037 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:11:00.637 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:11:00.0, currentDate=Mon May 15 14:11:00 CST 2023 2023-05-15 14:11:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:11:00.0} 2023-05-15 14:11:00.640 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:11:00.640 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:00.640 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:00.645 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:11:00.0, currentDate=Mon May 15 14:11:00 CST 2023 2023-05-15 14:11:00.645 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:11:00.0} 2023-05-15 14:11:00.655 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:11:00.655 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:00.655 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:00.660 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:11:00.0, currentDate=Mon May 15 14:11:00 CST 2023 2023-05-15 14:11:00.660 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:11:00.0} 2023-05-15 14:11:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:11:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:00.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:31.0, nextRun=2023-05-15 14:11:00.0, currentDate=Mon May 15 14:11:00 CST 2023 2023-05-15 14:11:00.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:10:31.0, nextRunTime=2023-05-15 14:11:00.0} 2023-05-15 14:11:00.686 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:11:00.686 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:00.686 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:00.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:11:00.0, currentDate=Mon May 15 14:11:00 CST 2023 2023-05-15 14:11:00.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:11:00.0} 2023-05-15 14:11:00.697 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:11:00.697 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:00.697 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:00.784 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131974, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:00.786 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131974, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:11:00.782+08:00, endTime=null} 2023-05-15 14:11:00.795 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131975, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:00.797 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131975, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:11:00.793+08:00, endTime=null} 2023-05-15 14:11:00.805 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ed16cb482772] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131974, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116362, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:00.782+08:00, sub_task_instance_id=131974}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:00.807 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131973, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:00.809 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131973, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:11:00.804+08:00, endTime=null} 2023-05-15 14:11:00.819 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131975, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116363, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:00.793+08:00, sub_task_instance_id=131975}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:00.822 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131976, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:00.823 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131976, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:11:00.818+08:00, endTime=null} 2023-05-15 14:11:00.825 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ed16cb482772] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:11:00.826 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ed16cb482772] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:11:00.829 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ad3f27aa7faa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131973, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116361, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:00.804+08:00, sub_task_instance_id=131973}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:00.830 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,ed16cb482772] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131974, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:00.833 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131977, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:00.834 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131977, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:11:00.831+08:00, endTime=null} 2023-05-15 14:11:00.835 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:11:00.835 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:11:00.835 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:11:00.835 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:11:00.835 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:11:00.838 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ad3f27aa7faa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131973, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:00.840 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:11:00.840 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:11:00.840 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:11:00.840 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,8dfd41bc13a8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131976, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116364, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:00.818+08:00, sub_task_instance_id=131976}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:00.840 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,8dfd41bc13a8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:11:00.841 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:11:00.841 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:11:00.841 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:11:00.841 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:11:00.842 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:11:00.843 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:11:00.847 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,3242009a50ad] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131975, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:00.851 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a1198b217d5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131977, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116365, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:00.830+08:00, sub_task_instance_id=131977}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:11:00.858 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:11:00.866 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:11:00.866 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:11:00.868 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:11:00.871 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:11:00.872 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a1198b217d5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131977, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:00.873 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:11:00.873 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.873 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.876 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.877 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.877 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:11:00.877 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.878 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.879 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:11:00.880 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:11:00.881 INFO 11 --- [pool-3246584-thread-1,fb27d19334114ed7,ac81d6631cf1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:11:00.881 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-2,fb27d19334114ed7,ffd7f595b466] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-3,fb27d19334114ed7,cb27b3eb5505] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.882 INFO 11 --- [pool-3246584-thread-4,fb27d19334114ed7,624aeef9c092] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:11:00.883 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:00.883 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:11:00.885 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.885 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.885 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:11:00.887 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:00.887 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:00.888 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:11:00.890 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:11:00.890 INFO 11 --- [pool-3246584-thread-5,fb27d19334114ed7,c3707d52e888] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:11:00.891 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:11:00.891 INFO 11 --- [pool-3246584-thread-6,fb27d19334114ed7,eaba27eea710] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:11:00.892 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,8dfd41bc13a8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:11:00.896 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,8dfd41bc13a8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131976, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:01.332 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a94f7e1f0770474e,fd54a783a11a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=a94f7e1f0770474e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:01.337 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a94f7e1f0770474e,fd54a783a11a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=a94f7e1f0770474e, duration=6 ms] 2023-05-15 14:11:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,c53482e02019] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131120, interval=60 2023-05-15 14:11:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,bf29fbdf5c72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131120, interval=60 2023-05-15 14:11:01.523 INFO 11 --- [pool-prometheus-minute-schedule-2,,c53482e02019] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:11:01.523 INFO 11 --- [pool-prometheus-minute-schedule-2,,c53482e02019] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131061, count=8, elapsed=1682446930462 2023-05-15 14:11:01.524 INFO 11 --- [pool-prometheus-minute-schedule-1,,bf29fbdf5c72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:11:01.524 INFO 11 --- [pool-prometheus-minute-schedule-1,,bf29fbdf5c72] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131061, count=6, elapsed=1682446930463 2023-05-15 14:11:01.760 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116361, status:SUCCESS 2023-05-15 14:11:01.761 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116361, status:SUCCESS 2023-05-15 14:11:01.770 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116362, status:SUCCESS 2023-05-15 14:11:01.771 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116362, status:SUCCESS 2023-05-15 14:11:01.781 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116363, status:SUCCESS 2023-05-15 14:11:01.782 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116363, status:SUCCESS 2023-05-15 14:11:01.791 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116364, status:SUCCESS 2023-05-15 14:11:01.792 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116364, status:SUCCESS 2023-05-15 14:11:01.801 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116365, status:SUCCESS 2023-05-15 14:11:01.802 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116365, status:SUCCESS 2023-05-15 14:11:02.936 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:02.937 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:04.714 INFO 11 --- [pool-ocp-schedules-4,,66367c7e01f0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:11:04.721 INFO 11 --- [pool-ocp-schedules-4,,66367c7e01f0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:11:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:04.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:04.830526+08:00, lastProcessTime=2023-05-15T14:10:54.829925+08:00 2023-05-15 14:11:04.860 INFO 11 --- [alarm-detect-9,,b8642371e41b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131044 ~ 1684131054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131050=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:04.895 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:04.830526+08:00 2023-05-15 14:11:05.749 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-05-15 14:11:05.835 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:06.0, nextRun=2023-05-15 14:11:05.0, currentDate=Mon May 15 14:11:05 CST 2023 2023-05-15 14:11:05.836 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:10:06.0, nextRunTime=2023-05-15 14:11:05.0} 2023-05-15 14:11:05.840 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:11:05.840 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:05.840 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:05.914 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131978, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:05.916 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131978, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:11:05.911+08:00, endTime=null} 2023-05-15 14:11:05.937 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:05.938 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:05.940 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1d6fbf91ce6d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131978, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116366, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:05.911+08:00, sub_task_instance_id=131978}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:05.966 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,6bad3debe7cd] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:11:05.966 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,11f9b1686159] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:11:05.967 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,cc10acd917ad] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:11:06.004 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,6bad3debe7cd] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:11:06.010 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,11f9b1686159] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:11:06.011 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,cc10acd917ad] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:11:06.022 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,1d6fbf91ce6d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131978, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:06.121 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:06.132 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:06.134 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:06.136 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:06.897 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116366, status:SUCCESS 2023-05-15 14:11:06.898 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116366, status:SUCCESS 2023-05-15 14:11:07.084 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:07.862 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,3f842772bee84bae,b7550893f87f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=3f842772bee84bae, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:11:07.863 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,3f842772bee84bae,b7550893f87f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=3f842772bee84bae, duration=2 ms] 2023-05-15 14:11:08.271 INFO 11 --- [pool-overview-container-8,,aaf8ef3c9e49] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:11:08.271 INFO 11 --- [pool-overview-container-8,,aaf8ef3c9e49] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:08.519 INFO 11 --- [pool-scanner--15975914-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:11:08.887 INFO 11 --- [pool-ocp-schedules-6,,531e500319fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:08.939 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:08.940 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:09.949 INFO 11 --- [pool-ocp-schedules-19,,b6823a3402a7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40060} 2023-05-15 14:11:09.949 INFO 11 --- [pool-ocp-schedules-19,,b6823a3402a7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:11:10.489 INFO 11 --- [pool-ocp-schedules-12,,45e60ec6a637] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:11:10.496 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,0d9992f15f2c4eef,bb76f3969b49] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=0d9992f15f2c4eef, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:10.502 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,0d9992f15f2c4eef,bb76f3969b49] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=0d9992f15f2c4eef, duration=6 ms] 2023-05-15 14:11:11.940 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:11.941 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:11.944 INFO 11 --- [pool-ocp-schedules-19,,ecb40b607372] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:11.944 INFO 11 --- [pool-ocp-schedules-19,,ecb40b607372] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:12.453 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:14.573 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131000, nodeCount=344, affectRows=344 2023-05-15 14:11:14.574 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131000, nodeCount=346, affectRows=346 2023-05-15 14:11:14.575 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131000, nodeCount=1000, affectRows=1000 2023-05-15 14:11:14.577 INFO 11 --- [pool-metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131000, nodeCount=344, validCount=344 2023-05-15 14:11:14.577 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131000, nodeCount=346, validCount=346 2023-05-15 14:11:14.583 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131000, nodeCount=1000, validCount=1000 2023-05-15 14:11:14.592 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131000, nodeCount=1000, affectRows=1000 2023-05-15 14:11:14.600 INFO 11 --- [pool-metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131000, nodeCount=1000, validCount=1000 2023-05-15 14:11:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:14.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:14.829376+08:00, lastProcessTime=2023-05-15T14:11:04.830526+08:00 2023-05-15 14:11:14.869 INFO 11 --- [alarm-detect-6,,9f1385006e9e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131054 ~ 1684131064 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131060=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:14.904 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:14.829376+08:00 2023-05-15 14:11:14.941 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:14.942 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:16.123 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:16.134 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:16.137 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:16.139 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:17.087 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:11:17.633 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:17.634 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:17.634 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:17.634 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.635 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.637 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:11:17.668 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:17.668 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:17.668 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.669 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:17.674 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:17.674 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.675 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:11:17.677 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:11:17.677 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:11:17.677 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:11:17.677 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:11:17.683 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:11:17.683 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.683 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:11:17.683 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.684 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:11:17.684 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:17.684 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:11:17.685 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:11:17.685 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:11:17.705 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:09:02, start=2023-05-15T14:05:47, end=2023-05-15T14:08:17, updates=102, granule=120000ms, delay=195000ms 2023-05-15 14:11:17.706 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:08:48, start=2023-05-15T14:05:47, end=2023-05-15T14:08:17, updates=98, granule=120000ms, delay=181000ms 2023-05-15 14:11:17.707 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:09:02, target=2023-05-15T14:08:17, granule=120000ms, delay=45000ms) 2023-05-15 14:11:17.708 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:08:17, target=2023-05-15T14:05:17, granule=600000ms, delay=180000ms) 2023-05-15 14:11:17.708 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:08:48, target=2023-05-15T14:08:17, granule=120000ms, delay=31000ms) 2023-05-15 14:11:17.708 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:08:17, target=2023-05-15T14:03:17, granule=600000ms, delay=300000ms) 2023-05-15 14:11:17.712 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:08:48, start=2023-05-15T14:05:47, end=2023-05-15T14:08:17, updates=133, granule=120000ms, delay=181000ms 2023-05-15 14:11:17.713 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:08:48, target=2023-05-15T14:08:17, granule=120000ms, delay=31000ms) 2023-05-15 14:11:17.713 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:08:17, target=2023-05-15T14:03:17, granule=600000ms, delay=300000ms) 2023-05-15 14:11:17.713 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=126ms, period=120000ms 2023-05-15 14:11:17.942 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:17.943 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:18.270 INFO 11 --- [pool-overview-container-5,,e7170777ede7] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:18.518 INFO 11 --- [pool-scanner--15976014-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:18.909 INFO 11 --- [pool-ocp-schedules-20,,baf49686dffe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:20.285 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:20.0, nextRun=2023-05-15 14:11:20.0, currentDate=Mon May 15 14:11:20 CST 2023 2023-05-15 14:11:20.285 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:10:20.0, nextRunTime=2023-05-15 14:11:20.0} 2023-05-15 14:11:20.289 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:11:20.289 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:20.289 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:20.374 INFO 11 --- [pool-ocp-schedules-20,,de88afee084e] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:11:20.375 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:11:20.375 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:11:20.375 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:11:20.375 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:11:20.375 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:11:20.382 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:11:20.382 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:11:20.382 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:11:20.382 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:11:20.382 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:11:20.411 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:11:20.411 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:11:20.412 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:11:20.413540397+08:00 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:11:20.413480854+08:00 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:11:20.414+08:00 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:11:20.414+08:00 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:11:20.413981624+08:00 2023-05-15 14:11:20.414 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:11:20.414+08:00 2023-05-15 14:11:20.416 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:11:20.415870016+08:00 2023-05-15 14:11:20.416 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:11:20.416+08:00 2023-05-15 14:11:20.416 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:11:20.416042562+08:00 2023-05-15 14:11:20.416 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:11:20.416+08:00 2023-05-15 14:11:20.417 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:11:20.414+08:00 and status AVAILABLE 2023-05-15 14:11:20.418 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:11:20.414+08:00 and status AVAILABLE 2023-05-15 14:11:20.419 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:11:20.414+08:00 and status AVAILABLE 2023-05-15 14:11:20.419 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:11:20.416+08:00 and status AVAILABLE 2023-05-15 14:11:20.420 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:11:20.416+08:00 and status AVAILABLE 2023-05-15 14:11:20.433 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:11:20.433 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:11:20.433 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:11:20.434 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:11:20.435 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:11:20.442 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131080 0 0 , error= 2023-05-15 14:11:20.457 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131080 0 0 , error= 2023-05-15 14:11:20.457 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131080 0 0 , error= 2023-05-15 14:11:20.461 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:11:20.456+08:00 2023-05-15 14:11:20.464 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131080 0 0 , error= 2023-05-15 14:11:20.464 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:11:20.464+08:00 2023-05-15 14:11:20.464 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:11:20.464+08:00 2023-05-15 14:11:20.464 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:11:20.464+08:00 2023-05-15 14:11:20.470 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:11:20.464+08:00 and status AVAILABLE 2023-05-15 14:11:20.470 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:11:20.456+08:00 and status AVAILABLE 2023-05-15 14:11:20.470 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:11:20.464+08:00 and status AVAILABLE 2023-05-15 14:11:20.470 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:11:20.464+08:00 and status AVAILABLE 2023-05-15 14:11:20.498 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:11:20.498 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:11:20.498 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:11:20.498 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:11:20.510 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:11:20.510 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:11:20.511 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:11:20.511 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:11:20.537 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:11:20.538 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:11:20.542 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:11:20.543 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:11:20.586 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:11:20.587 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:11:20.594 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:11:20.598 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:11:20.600 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:11:20.610 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:11:20.617 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:11:20.622 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:11:20.629 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:11:20.639 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:11:20.642 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:11:20.661 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:11:20.662 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:11:20.676 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:11:20.676 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:11:20.681 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:11:20.692 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:11:20.695 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:11:20.698 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:11:20.711 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:11:20.714 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:11:20.740 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:11:20.744 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:11:20.756 INFO 11 --- [pool-Check host status12,,8bfe93a18e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:11:20.758 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:11:20.766 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:11:20.775 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:11:20.779 INFO 11 --- [pool-Check host status26,,c030826d2c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:11:20.790 INFO 11 --- [pool-Check host status10,,aa2844af19c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:11:20.802 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:11:20.885 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:11:20.902 INFO 11 --- [pool-Check host status22,,6ee19a1d70bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:11:20.943 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:20.945 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:21.153 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131979, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:21.183 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131979, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:11:21.149+08:00, endTime=null} 2023-05-15 14:11:21.236 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a9c259164488] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131979, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116367, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:21.149+08:00, sub_task_instance_id=131979}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:21.238 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a9c259164488] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:11:21.238 INFO 11 --- [ForkJoinPool-66071-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:11:21.238 INFO 11 --- [ForkJoinPool-66071-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:11:21.243 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,a9c259164488] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131979, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:21.393 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131081 0 0 , error= 2023-05-15 14:11:21.396 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:11:21.396+08:00 2023-05-15 14:11:21.401 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:11:21.396+08:00 and status AVAILABLE 2023-05-15 14:11:21.404 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:11:21.413 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:11:21.436 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:11:21.481 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:11:21.493 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:11:21.537 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:11:21.599 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:11:21.613 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:11:21.636 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:11:21.686 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:11:21.701 INFO 11 --- [pool-Check host status9,,57caa98df282] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:11:21.721 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:11:21.722 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:11:21.722 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:11:21.722 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:11:21.722 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:11:21.727 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:11:21.728 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:11:21.728 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:11:21.728 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:11:21.729 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:11:21.751 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:11:21.753 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:11:21.754 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:11:21.754 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:11:21.757 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status25,,ea6a17976d4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:11:21.771 INFO 11 --- [pool-Check host status16,,5a7be94f05ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:11:21.775 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:11:21.775 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:11:21.775 INFO 11 --- [pool-Check host status15,,d889d0583dec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:11:21.776 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:11:21.776 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:11:21.776 INFO 11 --- [pool-Check host status24,,22efef8ef6a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:11:21.780 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:11:21.780 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:11:21.780 INFO 11 --- [pool-Check host status18,,bbf5c4075e44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:11:21.843 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:21.848 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:21.858 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:22.253 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116367, status:SUCCESS 2023-05-15 14:11:22.254 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116367, status:SUCCESS 2023-05-15 14:11:22.533 INFO 11 --- [pool-scanner--15976015-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:22.554 INFO 11 --- [pool-scanner--15976016-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:22.577 INFO 11 --- [pool-scanner--15976017-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:22.600 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:22.605 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:22.610 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,40a2a3ecf4624393,6e683cb79ea5] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObIDCRegionInfo&User_ID=admin&UID=alibaba&ObRegion=obcluster&version=2&ObRegionId=1, client=10.xxx.xx.10, traceId=40a2a3ecf4624393, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObIDCRegionInfo(String,Long,String,Long,Integer), args=obcluster,1,2,] 2023-05-15 14:11:22.612 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,40a2a3ecf4624393,6e683cb79ea5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=40a2a3ecf4624393, duration=2 ms] 2023-05-15 14:11:22.648 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:22.651 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:22.670 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:22.674 INFO 11 --- [pool-ocp-schedules-17,,69f8ba09b0b8] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:23.358 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,d5582d29d4754a38,159378b34705] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=d5582d29d4754a38, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:23.364 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,d5582d29d4754a38,159378b34705] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=d5582d29d4754a38, duration=6 ms] 2023-05-15 14:11:23.945 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:23.946 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:24.773 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,45e441b8c1274de6,188d853a3055] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=45e441b8c1274de6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:24.778 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,45e441b8c1274de6,188d853a3055] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=45e441b8c1274de6, duration=5 ms] 2023-05-15 14:11:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:24.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:24.832456+08:00, lastProcessTime=2023-05-15T14:11:14.829376+08:00 2023-05-15 14:11:24.858 INFO 11 --- [alarm-detect-13,,e630021b9bc6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131014 ~ 1684131074 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389457.988, values={1684131060=389457.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:11:24.858 INFO 11 --- [alarm-detect-8,,5b2bff87ed98] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131064 ~ 1684131074 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131070=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:24.859 INFO 11 --- [alarm-detect-13,,e630021b9bc6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131014 ~ 1684131074 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994255.358, values={1684131060=994255.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:24.861 INFO 11 --- [alarm-detect-13,,e630021b9bc6] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:11:24.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:24.832456+08:00 2023-05-15 14:11:25.477 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:26.0, nextRun=2023-05-15 14:11:25.0, currentDate=Mon May 15 14:11:25 CST 2023 2023-05-15 14:11:25.477 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:10:26.0, nextRunTime=2023-05-15 14:11:25.0} 2023-05-15 14:11:25.488 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:11:25.488 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:25.488 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:26.126 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:26.137 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:26.139 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:26.141 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:26.266 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131980, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:26.271 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131980, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:11:26.264+08:00, endTime=null} 2023-05-15 14:11:26.295 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131980, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116368, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:26.264+08:00, sub_task_instance_id=131980}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:11:26.343 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:11:26.344 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:11:26.345 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:11:26.346 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:11:26.346 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:11:26.352 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,94985590fb9a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131980, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:26.946 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:26.947 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:27.090 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:27.360 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116368, status:SUCCESS 2023-05-15 14:11:27.362 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116368, status:SUCCESS 2023-05-15 14:11:27.455 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:28.272 INFO 11 --- [pool-overview-container-3,,ccfa64d8eb8e] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:11:28.272 INFO 11 --- [pool-overview-container-3,,ccfa64d8eb8e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:28.522 INFO 11 --- [pool-scanner--15976115-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:28.941 INFO 11 --- [pool-ocp-schedules-2,,04c9d455f9d0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:29.948 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:29.949 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:30.503 INFO 11 --- [pool-ocp-schedules-18,,762c70166700] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:11:30.663 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:31.0, nextRun=2023-05-15 14:11:30.0, currentDate=Mon May 15 14:11:30 CST 2023 2023-05-15 14:11:30.663 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:10:31.0, nextRunTime=2023-05-15 14:11:30.0} 2023-05-15 14:11:30.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:11:30.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:30.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:30.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:30.683 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,ba49cb26d11c4b95,c171c212da94] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ba49cb26d11c4b95, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:30.684 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:01.0, nextRun=2023-05-15 14:11:30.0, currentDate=Mon May 15 14:11:30 CST 2023 2023-05-15 14:11:30.684 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:11:01.0, nextRunTime=2023-05-15 14:11:30.0} 2023-05-15 14:11:30.687 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,ba49cb26d11c4b95,c171c212da94] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ba49cb26d11c4b95, duration=4 ms] 2023-05-15 14:11:30.688 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:11:30.688 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:11:30.688 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:11:31.368 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131982, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:31.370 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131982, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:11:31.366+08:00, endTime=null} 2023-05-15 14:11:31.381 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131983, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:31.383 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131983, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:11:31.378+08:00, endTime=null} 2023-05-15 14:11:31.393 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131981, operation:EXECUTE, state: RUNNING 2023-05-15 14:11:31.393 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131982, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116369, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:31.365+08:00, sub_task_instance_id=131982}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:11:31.395 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131981, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:11:31.391+08:00, endTime=null} 2023-05-15 14:11:31.403 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,aba90a7c5f60] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131983, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116370, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:31.378+08:00, sub_task_instance_id=131983}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:11:31.403 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,aba90a7c5f60] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:11:31.413 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131981, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116369, task_operation=execute, latest_execution_start_time=2023-05-15T14:11:31.391+08:00, sub_task_instance_id=131981}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:11:31.426 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:11:31.426 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:11:31.426 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:11:31.427 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:11:31.427 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:11:31.427 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:11:31.438 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.438 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.438 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.438 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.438 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.439 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.440 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:11:31.441 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.441 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.441 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.441 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:11:31.441 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.442 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.442 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-2,fb27d19334114ed7,1a7e21eac7f3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-1,fb27d19334114ed7,f734b90ddbba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.443 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:11:31.444 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:11:31.444 INFO 11 --- [pool-3246742-thread-4,fb27d19334114ed7,5f244befaa31] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:11:31.444 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.444 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.445 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.445 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.445 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.445 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.446 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.446 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.446 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:11:31.447 INFO 11 --- [pool-3246742-thread-3,fb27d19334114ed7,eacc8ffdc564] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:11:31.451 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.451 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.451 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.451 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.451 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:11:31.452 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:11:31.452 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.452 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.452 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.453 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.453 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.453 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.454 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:11:31.455 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:11:31.455 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:11:31.455 INFO 11 --- [pool-3246742-thread-5,fb27d19334114ed7,4eb3209475b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:11:31.455 INFO 11 --- [pool-3246742-thread-6,fb27d19334114ed7,fb186c87f801] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:11:31.456 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,aba90a7c5f60] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:11:31.460 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.460 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.461 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,aba90a7c5f60] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131983, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:31.461 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.464 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.465 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.465 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.466 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.470 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.470 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.471 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:11:31.473 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.473 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.473 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.475 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.475 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.476 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.483 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.483 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.484 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.489 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.490 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.498 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.498 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.498 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.508 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.508 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.509 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:11:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.510 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.519 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.519 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.520 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:11:31.528 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.528 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.529 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.535 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.535 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.536 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:11:31.541 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:11:31.547 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.547 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.547 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:11:31.557 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:11:31.571 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.572 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.573 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.573 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.574 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.576 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.576 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.577 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.578 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.578 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.578 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.578 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.578 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.579 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.591 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.592 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.592 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.592 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.592 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.593 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.593 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.594 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.599 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.600 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.601 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.601 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.602 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.606 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.606 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.607 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.614 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.615 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.615 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.615 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.615 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.618 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.618 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.618 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.639 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.639 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.640 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.640 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.640 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.641 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.641 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.641 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.645 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.646 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.647 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.648 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.652 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.653 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.653 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.658 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.659 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.659 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.659 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.659 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.660 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.660 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.660 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.671 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.672 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.672 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.672 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.672 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.673 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.673 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.673 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.674 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.674 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.674 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.674 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.675 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.682 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.682 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.683 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.683 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:11:31.687 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:11:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.691 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.692 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.692 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.692 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.693 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:11:31.701 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.701 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.701 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.701 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.701 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.702 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.702 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.702 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:11:31.703 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:11:31.705 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:11:31.707 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.708 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.709 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.709 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.709 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.715 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.715 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.716 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.720 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:31.722 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.722 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.722 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.728 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.729 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:11:31.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:31.730 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.731 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:11:31.731 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.733 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:11:31.733 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.733 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.733 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:11:31.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.742 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:11:31.742 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:11:31.742 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:11:31.742 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.743 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:11:31.744 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.744 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.744 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:11:31.745 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,525ce28e9439] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131982, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:31.757 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.759 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:11:31.760 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:31.760 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:31.760 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:11:31.773 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:11:31.775 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:11:31.775 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:11:31.775 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:11:31.780 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c647fa46f2f6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131981, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:11:31.949 INFO 11 --- [pool-ocp-schedules-10,,8a537ad4105e] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:31.949 INFO 11 --- [pool-ocp-schedules-10,,8a537ad4105e] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:32.066 INFO 11 --- [pool-ocp-schedules-5,,dbab02150d0b] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:11:32.069 INFO 11 --- [pool-ocp-schedules-5,,dbab02150d0b] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:11:32.075 INFO 11 --- [pool-ocp-schedules-4,,d7d8b65d5089] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:11:32.078 INFO 11 --- [pool-ocp-schedules-4,,d7d8b65d5089] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:11:32.089 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.089 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.089 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.090 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.090 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.091 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.091 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.091 INFO 11 --- [ForkJoinPool-66072-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:11:32.094 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.095 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.095 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.096 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.096 INFO 11 --- [pool-ocp-schedules-1,,726bcdd3ca8e] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:11:32.098 INFO 11 --- [pool-ocp-schedules-1,,726bcdd3ca8e] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:11:32.100 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.100 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.101 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.104 INFO 11 --- [ForkJoinPool-66072-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:11:32.123 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.123 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.124 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.124 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.125 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.125 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.126 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:11:32.142 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.142 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.142 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.143 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.147 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.148 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.148 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:11:32.182 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:32.183 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.183 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.183 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.184 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.185 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.185 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.185 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:11:32.186 WARN 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:11:32.186 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.186 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.187 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:11:32.188 INFO 11 --- [ForkJoinPool-66074-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:11:32.188 INFO 11 --- [ForkJoinPool-66074-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:11:32.188 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:11:32.195 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.196 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:11:32.204 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.212 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:32.213 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.213 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.213 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.214 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.217 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:32.217 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.218 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.218 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.218 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.218 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.218 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.219 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:11:32.219 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:11:32.219 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.220 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.223 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.223 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.224 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:11:32.226 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:11:32.230 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:11:32.230 INFO 11 --- [ForkJoinPool-66074-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:11:32.241 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.241 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:11:32.256 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.256 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:11:32.272 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.272 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:11:32.292 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.292 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:11:32.310 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.310 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:11:32.328 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:11:32.346 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:11:32.347 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.348 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.348 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.348 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.353 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:11:32.353 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:11:32.354 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:11:32.375 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:11:32.375 INFO 11 --- [ForkJoinPool-66074-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:11:32.466 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116369, status:SUCCESS 2023-05-15 14:11:32.468 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116369, status:SUCCESS 2023-05-15 14:11:32.478 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116370, status:SUCCESS 2023-05-15 14:11:32.479 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116370, status:SUCCESS 2023-05-15 14:11:32.949 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:32.951 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:34.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:34.829453+08:00, lastProcessTime=2023-05-15T14:11:24.832456+08:00 2023-05-15 14:11:34.850 INFO 11 --- [alarm-detect-10,,83317c527437] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684131074 ~ 1684131084 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=106.0, values={1684131080=106.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:11:34.853 INFO 11 --- [alarm-detect-10,,83317c527437] c.o.o.s.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, currentState=Pending 2023-05-15 14:11:34.853 INFO 11 --- [alarm-detect-4,,ea2763303cb0] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131074 ~ 1684131084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131080=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:34.864 INFO 11 --- [alarm-detect-13,,aa66d2a93062] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131024 ~ 1684131084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389457.938, values={1684131060=389457.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:11:34.865 INFO 11 --- [alarm-detect-13,,aa66d2a93062] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131024 ~ 1684131084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994255.107, values={1684131060=994255.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:34.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:34.829453+08:00 2023-05-15 14:11:35.951 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:35.952 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:36.128 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:36.138 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:36.141 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:36.143 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:37.094 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:38.275 INFO 11 --- [pool-overview-container-4,,4237121a9e27] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:38.522 INFO 11 --- [pool-scanner--15976226-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:38.940 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,465cd4f4897c4d3a,3221e02f6c3c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=465cd4f4897c4d3a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:38.946 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,465cd4f4897c4d3a,3221e02f6c3c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=465cd4f4897c4d3a, duration=7 ms] 2023-05-15 14:11:38.953 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:38.954 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:38.962 INFO 11 --- [pool-ocp-schedules-11,,1afcd84146cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:41.895 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,582dcf0d13704904,f59446eeb53b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=582dcf0d13704904, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:11:41.897 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,582dcf0d13704904,f59446eeb53b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=582dcf0d13704904, duration=3 ms] 2023-05-15 14:11:41.954 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:41.955 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:42.458 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:44.394 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,1d16251ec5084dba,7c761f7bfd10] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=1d16251ec5084dba, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:44.400 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,1d16251ec5084dba,7c761f7bfd10] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=1d16251ec5084dba, duration=6 ms] 2023-05-15 14:11:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:44.836 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:44.832288+08:00, lastProcessTime=2023-05-15T14:11:34.829453+08:00 2023-05-15 14:11:44.850 INFO 11 --- [alarm-detect-9,,5aa1fc301b8b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684131074 ~ 1684131084 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=106.0, values={1684131080=106.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:11:44.855 INFO 11 --- [alarm-detect-2,,9695fef27449] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131084 ~ 1684131094 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131090=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:44.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:44.832288+08:00 2023-05-15 14:11:44.955 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:44.956 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:45.515 INFO 11 --- [pool-ocp-schedules-6,,97a814dcb984] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:11:46.131 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:46.141 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:46.143 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:46.145 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:47.097 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:47.956 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:47.958 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:48.271 INFO 11 --- [pool-overview-container-1,,eda2b01b6bcf] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:48.517 INFO 11 --- [pool-scanner--15976326-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:48.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,bd320343adf8467f,a3f87c7c5300] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=bd320343adf8467f, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:48.708 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,bd320343adf8467f,a3f87c7c5300] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=bd320343adf8467f, duration=6 ms] 2023-05-15 14:11:48.984 INFO 11 --- [pool-ocp-schedules-11,,77104dca71ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:50.958 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:50.959 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:51.848 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:51.853 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:51.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:51.954 INFO 11 --- [pool-ocp-schedules-15,,a5f6f3740781] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:51.954 INFO 11 --- [pool-ocp-schedules-15,,a5f6f3740781] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:11:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,953a893d0152] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2700,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40060} 2023-05-15 14:11:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,953a893d0152] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:11:53.959 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:53.961 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:11:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:11:54.830935+08:00, lastProcessTime=2023-05-15T14:11:44.832288+08:00 2023-05-15 14:11:54.846 INFO 11 --- [alarm-detect-4,,1d0f72117a73] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_cpu_percent_over_threshold detect time range 1684131074 ~ 1684131084 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={ob_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_cpu_percent, match=true, value=106.0, values={1684131080=106.0}, valueCount=1, labels={app=OB, host=10.xxx.xx.11, host_name=obdb1, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, obzone=zone1, svr_ip=10.xxx.xx.11, tenant_name=sys})]}, matched=true, message=null, level=3) 2023-05-15 14:11:54.857 INFO 11 --- [alarm-detect-12,,a159a6542b4a] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131094 ~ 1684131104 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131100=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:11:54.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:11:54.830935+08:00 2023-05-15 14:11:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,76c7e8d24d53] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:11:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,44dd68b36407] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:11:56.134 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:56.167 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:56.188 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:56.202 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:56.961 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:56.962 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:11:57.100 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:57.461 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:11:58.275 INFO 11 --- [pool-overview-container-6,,65e818f3ea63] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:11:58.539 INFO 11 --- [pool-scanner--15976420-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:11:58.687 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d6c5e9acbbd74118,f7e65a1d0547] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=d6c5e9acbbd74118, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:11:58.693 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,d6c5e9acbbd74118,f7e65a1d0547] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=d6c5e9acbbd74118, duration=6 ms] 2023-05-15 14:11:59.005 INFO 11 --- [pool-ocp-schedules-11,,20b583bc811c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:11:59.962 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:11:59.963 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:12:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:12:00.000 INFO 11 --- [pool-ocp-schedules-15,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131120 2023-05-15 14:12:00.000 INFO 11 --- [pool-ocp-schedules-16,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:12:00.000 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:12:00.001 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:12:00.013 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:12:00.013 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:12:00.014 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:12:00.014 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:12:00.018 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:12:00.019 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:12:00.019 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:12:00.019 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:12:00.043 INFO 11 --- [pool-ocp-schedules-11,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:12:00.057 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:12:00.066 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:12:00.066 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:12:00.066 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:12:00.066 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:12:00.066 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:12:00.074 WARN 11 --- [pool-ocp-schedules-16,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:12:00.074 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:12:00.563 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.563 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:12:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.580 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:12:00.703 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:01.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.703 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:11:01.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.707 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:12:00.707 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.707 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.712 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:01.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.712 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:11:01.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.724 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:12:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.725 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.730 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:01.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:11:01.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:12:00.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.748 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.755 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:31.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.755 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:11:31.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:12:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.760 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.771 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:01.0, nextRun=2023-05-15 14:12:00.0, currentDate=Mon May 15 14:12:00 CST 2023 2023-05-15 14:12:00.771 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:11:01.0, nextRunTime=2023-05-15 14:12:00.0} 2023-05-15 14:12:00.776 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:12:00.776 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:00.776 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:00.847 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131986, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.849 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131986, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:12:00.844+08:00, endTime=null} 2023-05-15 14:12:00.859 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131987, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.861 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131987, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:12:00.857+08:00, endTime=null} 2023-05-15 14:12:00.866 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,de454e871d42] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131986, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116372, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.844+08:00, sub_task_instance_id=131986}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.871 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131984, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.872 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:12:00.872 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:12:00.872 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:12:00.872 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:12:00.872 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:12:00.872 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131984, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:12:00.868+08:00, endTime=null} 2023-05-15 14:12:00.876 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,de454e871d42] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131986, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.877 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:12:00.877 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:12:00.877 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:12:00.877 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:12:00.877 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:12:00.878 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c63f36d0227b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131987, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116373, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.857+08:00, sub_task_instance_id=131987}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.883 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131985, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.885 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131985, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:12:00.881+08:00, endTime=null} 2023-05-15 14:12:00.893 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5220b0ba9d49] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131984, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116371, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.868+08:00, sub_task_instance_id=131984}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.894 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131990, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.896 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131990, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:12:00.892+08:00, endTime=null} 2023-05-15 14:12:00.896 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c63f36d0227b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:12:00.897 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c63f36d0227b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:12:00.899 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:12:00.899 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:12:00.901 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:12:00.901 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,c63f36d0227b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131987, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.901 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:12:00.902 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:12:00.903 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:12:00.904 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:12:00.904 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,b26a4ecea3c6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131985, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116371, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.881+08:00, sub_task_instance_id=131985}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.904 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131988, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.905 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:12:00.906 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131988, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:12:00.902+08:00, endTime=null} 2023-05-15 14:12:00.908 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5220b0ba9d49] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131984, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.910 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:12:00.910 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,e0fea2672e87] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131990, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116376, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.891+08:00, sub_task_instance_id=131990}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.911 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:12:00.913 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:12:00.914 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131989, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:00.914 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,b26a4ecea3c6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131985, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.915 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131989, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:12:00.912+08:00, endTime=null} 2023-05-15 14:12:00.920 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:12:00.920 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:12:00.920 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131988, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116374, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.902+08:00, sub_task_instance_id=131988}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.923 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,e0fea2672e87] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131990, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.928 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,1a563c07158c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131989, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116375, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:00.912+08:00, sub_task_instance_id=131989}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:00.929 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,1a563c07158c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:12:00.939 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:12:00.939 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:12:00.940 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:12:00.940 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:12:00.945 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2aefcd46478d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131988, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:12:00.945 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:12:00.958 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.959 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.960 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.960 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.960 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.960 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.960 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.961 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.962 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.963 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:12:00.963 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:00.963 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:12:00.963 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.963 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:12:00.964 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.964 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.964 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.965 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.965 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:12:00.965 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-4,fb27d19334114ed7,59ce33787e50] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-3,fb27d19334114ed7,18f4f0b0c8e3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-2,fb27d19334114ed7,485dfe7210fc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:12:00.966 INFO 11 --- [pool-3246911-thread-1,fb27d19334114ed7,15e1db50b037] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:12:00.967 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.968 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.968 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.968 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:00.970 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.970 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.971 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:12:00.972 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:00.972 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:00.973 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:12:00.974 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:12:00.974 INFO 11 --- [pool-3246911-thread-6,fb27d19334114ed7,31dc0b41b3e7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:12:00.978 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:12:00.978 INFO 11 --- [pool-3246911-thread-5,fb27d19334114ed7,5ed6f40ae1ed] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:12:00.979 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,1a563c07158c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:12:00.982 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116371, status:SUCCESS 2023-05-15 14:12:00.983 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,1a563c07158c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131989, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:00.983 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116371, status:SUCCESS 2023-05-15 14:12:00.991 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116372, status:SUCCESS 2023-05-15 14:12:00.992 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116372, status:SUCCESS 2023-05-15 14:12:00.999 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116373, status:SUCCESS 2023-05-15 14:12:01.000 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116373, status:SUCCESS 2023-05-15 14:12:01.008 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116374, status:SUCCESS 2023-05-15 14:12:01.009 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116374, status:SUCCESS 2023-05-15 14:12:01.017 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116375, status:SUCCESS 2023-05-15 14:12:01.018 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116375, status:SUCCESS 2023-05-15 14:12:01.027 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116376, status:SUCCESS 2023-05-15 14:12:01.028 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116376, status:SUCCESS 2023-05-15 14:12:01.228 WARN 11 --- [pool-ocp-schedules-16,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:12:01.229 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.210324795S 2023-05-15 14:12:01.229 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:12:01.345 INFO 11 --- [pool-ocp-schedules-16,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:12:01.411 INFO 11 --- [pool-ocp-schedules-16,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:12:01.449 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:12:01.449 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:12:01.449 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:12:01.449 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:12:01.449 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:12:01.450 INFO 11 --- [pool-ocp-schedules-16,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:12:01.459 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:12:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,aeb30e2f953e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131180, interval=60 2023-05-15 14:12:01.525 INFO 11 --- [pool-prometheus-minute-schedule-1,,fb31da0a9e3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131180, interval=60 2023-05-15 14:12:01.546 INFO 11 --- [pool-prometheus-minute-schedule-1,,fb31da0a9e3a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:12:01.546 INFO 11 --- [pool-prometheus-minute-schedule-1,,fb31da0a9e3a] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131121, count=6, elapsed=1682446990425 2023-05-15 14:12:01.546 INFO 11 --- [pool-prometheus-minute-schedule-2,,aeb30e2f953e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:12:01.547 INFO 11 --- [pool-prometheus-minute-schedule-2,,aeb30e2f953e] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131121, count=8, elapsed=1682446990426 2023-05-15 14:12:01.623 INFO 11 --- [pool-ocp-schedules-16,,] o.s.boot.SpringApplication : Started application in 1.61 seconds (JVM running for 600777.597) 2023-05-15 14:12:01.623 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.610219749S 2023-05-15 14:12:01.863 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:12:01.863 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:12:01.863 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:12:01.863 INFO 11 --- [pool-ocp-schedules-16,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:12:02.113 INFO 11 --- [pool-ocp-schedules-16,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:12:02.964 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:02.965 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:04.714 INFO 11 --- [pool-ocp-schedules-12,,9ee70232404c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:12:04.721 INFO 11 --- [pool-ocp-schedules-12,,9ee70232404c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:12:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:04.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:04.832463+08:00, lastProcessTime=2023-05-15T14:11:54.830935+08:00 2023-05-15 14:12:04.853 INFO 11 --- [alarm-detect-14,,a8cc386dcefe] c.o.o.s.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=tenant_cpu_percent_over_threshold, target=alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys:host=10.xxx.xx.11-3, activeAt=2023-05-15T14:11:34.841+08:00 2023-05-15 14:12:04.853 INFO 11 --- [alarm-detect-8,,8eac429ea615] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131104 ~ 1684131114 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.129, values={1684131110=246.129}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:04.854 INFO 11 --- [alarm-detect-14,,a8cc386dcefe] .o.o.s.a.c.d.AlarmRuleDetectStateManager : alarmType tenant_cpu_percent_over_threshold targets.size 10 was marked as not match 2023-05-15 14:12:04.891 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:04.832463+08:00 2023-05-15 14:12:05.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:06.0, nextRun=2023-05-15 14:12:05.0, currentDate=Mon May 15 14:12:05 CST 2023 2023-05-15 14:12:05.914 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:11:06.0, nextRunTime=2023-05-15 14:12:05.0} 2023-05-15 14:12:05.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:12:05.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:05.918 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:05.966 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:05.967 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:06.003 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131991, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:06.005 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131991, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:12:05.991+08:00, endTime=null} 2023-05-15 14:12:06.026 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,7ff3d892ee13] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131991, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116377, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:05.991+08:00, sub_task_instance_id=131991}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:06.051 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,ceb7572e9349] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:12:06.051 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,5f915fedf346] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:12:06.052 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,2f2050d00cf0] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:12:06.096 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,2f2050d00cf0] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:12:06.096 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,ceb7572e9349] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:12:06.100 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,5f915fedf346] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:12:06.118 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,7ff3d892ee13] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131991, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:06.147 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116377, status:SUCCESS 2023-05-15 14:12:06.148 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116377, status:SUCCESS 2023-05-15 14:12:06.167 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:06.188 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:06.202 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:06.204 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:07.103 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:08.273 INFO 11 --- [pool-overview-container-2,,5ced2856b010] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:12:08.274 INFO 11 --- [pool-overview-container-2,,5ced2856b010] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:08.519 INFO 11 --- [pool-scanner--15976513-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:12:08.967 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:08.968 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:09.026 INFO 11 --- [pool-ocp-schedules-5,,533d85e1d35d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:10.525 INFO 11 --- [pool-ocp-schedules-20,,c7ba9e54d716] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:12:11.959 INFO 11 --- [pool-ocp-schedules-9,,bb6493edf011] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:11.959 INFO 11 --- [pool-ocp-schedules-9,,bb6493edf011] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:11.968 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:11.969 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:12.463 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:14.627 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131060, nodeCount=1000, affectRows=1000 2023-05-15 14:12:14.632 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131060, nodeCount=1000, validCount=1000 2023-05-15 14:12:14.653 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131060, nodeCount=785, affectRows=785 2023-05-15 14:12:14.658 INFO 11 --- [pool-metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131060, nodeCount=785, validCount=785 2023-05-15 14:12:14.670 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131060, nodeCount=915, affectRows=915 2023-05-15 14:12:14.676 INFO 11 --- [pool-metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131060, nodeCount=915, validCount=915 2023-05-15 14:12:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:14.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:14.827664+08:00, lastProcessTime=2023-05-15T14:12:04.832463+08:00 2023-05-15 14:12:14.861 INFO 11 --- [alarm-detect-12,,a151af896c59] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131114 ~ 1684131124 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.175, values={1684131120=246.175}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:14.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:14.827664+08:00 2023-05-15 14:12:14.969 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:14.971 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:15.926 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a942cb73dc844051,0d1abb52b3fd] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=a942cb73dc844051, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:12:15.928 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a942cb73dc844051,0d1abb52b3fd] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=a942cb73dc844051, duration=2 ms] 2023-05-15 14:12:16.169 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:16.190 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:16.204 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:16.206 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:17.109 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:17.971 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:17.972 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:18.053 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,c3af21a1a5934c90,2fad7419bbc7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=c3af21a1a5934c90, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:18.059 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,c3af21a1a5934c90,2fad7419bbc7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=c3af21a1a5934c90, duration=7 ms] 2023-05-15 14:12:18.271 INFO 11 --- [pool-overview-container-4,,9624c4ef5220] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:18.292 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,76dd58cc21624061,1cd3cddc305a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=76dd58cc21624061, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:18.298 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,76dd58cc21624061,1cd3cddc305a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=76dd58cc21624061, duration=6 ms] 2023-05-15 14:12:18.523 INFO 11 --- [pool-scanner--15976613-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:19.048 INFO 11 --- [pool-ocp-schedules-6,,4fc868b11323] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:20.080 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,64ceacd8b3354703,253d0974a6b4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=64ceacd8b3354703, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:20.085 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,64ceacd8b3354703,253d0974a6b4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=64ceacd8b3354703, duration=6 ms] 2023-05-15 14:12:20.369 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:20.0, nextRun=2023-05-15 14:12:20.0, currentDate=Mon May 15 14:12:20 CST 2023 2023-05-15 14:12:20.369 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:11:20.0, nextRunTime=2023-05-15 14:12:20.0} 2023-05-15 14:12:20.373 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:12:20.373 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:20.373 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:20.972 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:20.973 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:21.244 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131992, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:21.246 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131992, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:12:21.241+08:00, endTime=null} 2023-05-15 14:12:21.266 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,aeb137df2ed1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131992, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116378, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:21.241+08:00, sub_task_instance_id=131992}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:21.268 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,aeb137df2ed1] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:12:21.268 INFO 11 --- [ForkJoinPool-66077-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:12:21.268 INFO 11 --- [ForkJoinPool-66077-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:12:21.275 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,aeb137df2ed1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131992, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:21.438 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116378, status:SUCCESS 2023-05-15 14:12:21.439 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116378, status:SUCCESS 2023-05-15 14:12:21.791 INFO 11 --- [pool-ocp-schedules-7,,b9b47c689a44] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:12:21.791 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:12:21.791 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:12:21.791 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:12:21.791 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:12:21.791 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:12:21.797 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:12:21.797 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:12:21.798 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:12:21.798 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:12:21.798 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:12:21.827 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:12:21.827 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:12:21.829653521+08:00 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:12:21.829581753+08:00 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:12:21.830+08:00 2023-05-15 14:12:21.830 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:12:21.830+08:00 2023-05-15 14:12:21.832 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:12:21.831785681+08:00 2023-05-15 14:12:21.832 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:12:21.831899922+08:00 2023-05-15 14:12:21.832 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:12:21.832+08:00 2023-05-15 14:12:21.832 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:12:21.832+08:00 2023-05-15 14:12:21.835 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:12:21.837 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:12:21.837150923+08:00 2023-05-15 14:12:21.837 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:12:21.837+08:00 2023-05-15 14:12:21.842 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:12:21.830+08:00 and status AVAILABLE 2023-05-15 14:12:21.849 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:12:21.850 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:12:21.832+08:00 and status AVAILABLE 2023-05-15 14:12:21.850 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:21.859 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:21.870 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131141 0 0 , error= 2023-05-15 14:12:21.870 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:12:21.830+08:00 and status AVAILABLE 2023-05-15 14:12:21.870 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:12:21.870+08:00 2023-05-15 14:12:21.872 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:21.873 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:12:21.832+08:00 and status AVAILABLE 2023-05-15 14:12:21.876 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:12:21.870+08:00 and status AVAILABLE 2023-05-15 14:12:21.876 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:12:21.877 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:12:21.837+08:00 and status AVAILABLE 2023-05-15 14:12:21.878 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:12:21.895 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:12:21.897 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131141 0 0 , error= 2023-05-15 14:12:21.897 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:12:21.897+08:00 2023-05-15 14:12:21.898 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:12:21.906 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131141 0 0 , error= 2023-05-15 14:12:21.919 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131141 0 0 , error= 2023-05-15 14:12:21.922 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:12:21.897+08:00 and status AVAILABLE 2023-05-15 14:12:21.922 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:12:21.922+08:00 2023-05-15 14:12:21.922 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:12:21.922+08:00 2023-05-15 14:12:21.923 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:12:21.925 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:12:21.961 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131141 0 0 , error= 2023-05-15 14:12:21.962 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:12:21.922+08:00 and status AVAILABLE 2023-05-15 14:12:21.962 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:12:21.922+08:00 and status AVAILABLE 2023-05-15 14:12:21.962 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:12:21.962+08:00 2023-05-15 14:12:21.963 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:12:21.964 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:12:21.965 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:12:21.967 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:12:21.962+08:00 and status AVAILABLE 2023-05-15 14:12:21.971 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:12:21.972 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:12:21.975 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:12:21.978 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:12:21.988 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:12:21.995 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:12:22.003 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:12:22.006 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:12:22.006 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:12:22.015 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:12:22.049 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:12:22.052 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:12:22.061 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:12:22.064 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:12:22.066 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:12:22.067 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:12:22.076 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:12:22.082 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:12:22.097 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:12:22.100 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:12:22.103 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:12:22.106 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:12:22.107 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:12:22.109 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:12:22.121 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:12:22.148 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:12:22.150 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:12:22.159 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:12:22.159 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:12:22.162 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:12:22.164 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:12:22.171 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:12:22.172 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:12:22.183 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:12:22.185 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:12:22.187 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:12:22.189 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:12:22.204 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:12:22.217 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:12:22.229 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:12:22.238 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:12:22.243 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:12:22.244 INFO 11 --- [pool-Check host status29,,358da4f44c9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:12:22.248 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:12:22.254 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:12:22.258 INFO 11 --- [pool-Check host status21,,e0f25ce0a6b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:12:22.263 INFO 11 --- [pool-Check host status7,,ce172f40576a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:12:22.268 INFO 11 --- [pool-Check host status4,,c0f34c4909f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:12:22.326 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:12:22.341 INFO 11 --- [pool-Check host status6,,44af11e2d2e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:12:22.359 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:12:22.360 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:12:22.360 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:12:22.360 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:12:22.360 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:12:22.367 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:12:22.367 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:12:22.370 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:12:22.370 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:12:22.370 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:12:22.395 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:12:22.398 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:12:22.399 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:12:22.401 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:12:22.402 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:12:22.416 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:12:22.416 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:12:22.416 INFO 11 --- [pool-Check host status23,,5aaab73d9ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:12:22.417 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:12:22.417 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:12:22.417 INFO 11 --- [pool-Check host status27,,6bf59222452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:12:22.418 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:12:22.418 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:12:22.418 INFO 11 --- [pool-Check host status11,,938c2be27c7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:12:22.421 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:12:22.422 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:12:22.422 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:12:22.422 INFO 11 --- [pool-Check host status5,,e8c614f36b2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:12:22.422 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:12:22.422 INFO 11 --- [pool-Check host status20,,4ce70eea77c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:12:22.715 INFO 11 --- [pool-scanner--15976614-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:22.732 INFO 11 --- [pool-scanner--15976615-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:22.749 INFO 11 --- [pool-scanner--15976616-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:22.768 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:22.774 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:22.809 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:22.812 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:22.829 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:22.832 INFO 11 --- [pool-ocp-schedules-5,,6fad072d12eb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:23.974 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:23.975 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:24.830125+08:00, lastProcessTime=2023-05-15T14:12:14.827664+08:00 2023-05-15 14:12:24.855 INFO 11 --- [alarm-detect-5,,dd54d76daf6c] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131124 ~ 1684131134 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.175, values={1684131130=246.175}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:24.856 INFO 11 --- [alarm-detect-8,,107ce9f99ebf] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131074 ~ 1684131134 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389517.988, values={1684131120=389517.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:12:24.857 INFO 11 --- [alarm-detect-8,,107ce9f99ebf] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131074 ~ 1684131134 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994315.358, values={1684131120=994315.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:24.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:24.830125+08:00 2023-05-15 14:12:25.535 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:25.0, nextRun=2023-05-15 14:12:25.0, currentDate=Mon May 15 14:12:25 CST 2023 2023-05-15 14:12:25.535 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:11:25.0, nextRunTime=2023-05-15 14:12:25.0} 2023-05-15 14:12:25.545 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:12:25.545 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:25.545 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:26.172 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:26.192 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:26.206 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:26.209 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:26.330 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131993, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:26.331 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131993, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:12:26.328+08:00, endTime=null} 2023-05-15 14:12:26.345 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131993, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116379, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:26.328+08:00, sub_task_instance_id=131993}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:12:26.385 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:12:26.386 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:12:26.387 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:12:26.388 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:12:26.388 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:12:26.394 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,51d433aae409] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131993, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:26.545 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116379, status:SUCCESS 2023-05-15 14:12:26.546 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116379, status:SUCCESS 2023-05-15 14:12:26.975 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:26.976 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:27.112 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:27.465 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:28.272 INFO 11 --- [pool-overview-container-8,,8bd4989e1f96] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:28.527 INFO 11 --- [pool-scanner--15976714-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:29.070 INFO 11 --- [pool-ocp-schedules-14,,0c05da51ff56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:29.976 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:29.978 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:30.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:11:31.0, nextRun=2023-05-15 14:12:30.0, currentDate=Mon May 15 14:12:30 CST 2023 2023-05-15 14:12:30.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:11:31.0, nextRunTime=2023-05-15 14:12:30.0} 2023-05-15 14:12:30.716 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:12:30.716 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:30.716 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:30.716 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:30.728 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:12:30.0, currentDate=Mon May 15 14:12:30 CST 2023 2023-05-15 14:12:30.728 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:12:30.0} 2023-05-15 14:12:30.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:12:30.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:12:30.731 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:12:31.408 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131994, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:31.410 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131994, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:12:31.405+08:00, endTime=null} 2023-05-15 14:12:31.420 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131995, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:31.422 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131995, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:12:31.418+08:00, endTime=null} 2023-05-15 14:12:31.430 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131994, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116380, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:31.404+08:00, sub_task_instance_id=131994}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:12:31.431 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131996, operation:EXECUTE, state: RUNNING 2023-05-15 14:12:31.434 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131996, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:12:31.429+08:00, endTime=null} 2023-05-15 14:12:31.439 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131995, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116380, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:31.418+08:00, sub_task_instance_id=131995}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:12:31.451 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ecc0bf11a5d6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131996, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116381, task_operation=execute, latest_execution_start_time=2023-05-15T14:12:31.429+08:00, sub_task_instance_id=131996}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:12:31.451 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ecc0bf11a5d6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:12:31.466 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:12:31.468 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.468 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.468 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.469 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.475 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.475 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.475 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:12:31.476 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.476 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.476 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.476 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.476 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.477 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.478 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.482 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.483 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.483 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.483 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.483 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.484 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.484 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.484 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.486 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.486 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.486 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:12:31.487 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.487 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.487 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.488 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.488 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.489 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:12:31.489 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:12:31.489 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.489 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.490 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.490 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.490 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:12:31.490 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:12:31.491 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.491 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.491 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:12:31.492 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:12:31.493 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.493 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.493 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:12:31.493 INFO 11 --- [pool-3247069-thread-4,fb27d19334114ed7,ff0b96e8de64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:12:31.494 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.495 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.495 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.495 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:12:31.495 INFO 11 --- [pool-3247069-thread-1,fb27d19334114ed7,98f6d3ac6d4a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:12:31.495 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:12:31.496 INFO 11 --- [pool-3247069-thread-5,fb27d19334114ed7,cca29be2d56f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:12:31.496 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:12:31.496 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.496 INFO 11 --- [pool-3247069-thread-2,fb27d19334114ed7,89c06f8e6a9a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:12:31.496 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:12:31.496 INFO 11 --- [pool-3247069-thread-6,fb27d19334114ed7,4db88598c88c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:12:31.497 INFO 11 --- [pool-3247069-thread-3,fb27d19334114ed7,b0a8d29bfe64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:12:31.498 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ecc0bf11a5d6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:12:31.499 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.499 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.499 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.500 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.501 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.502 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:12:31.502 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,ecc0bf11a5d6] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131996, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.506 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.507 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.507 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.507 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.508 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.518 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.518 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.518 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.522 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.522 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.523 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.527 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.527 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.528 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:12:31.535 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.535 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.536 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.538 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.538 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.538 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:12:31.549 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.549 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.549 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:12:31.557 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.558 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.559 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.561 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.561 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.561 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:12:31.571 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:12:31.576 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:12:31.590 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.590 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.591 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.591 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.591 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.599 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.599 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.599 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.604 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.605 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.605 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.605 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.606 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.609 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.610 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.610 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.610 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.610 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.610 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.611 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.611 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.611 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.611 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.611 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.630 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.630 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.631 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.631 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.631 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.631 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.632 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.632 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.632 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.633 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.633 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.633 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.633 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.638 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.639 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.640 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.647 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.647 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.648 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.648 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.648 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.649 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.649 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.649 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.651 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116381, status:SUCCESS 2023-05-15 14:12:31.652 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116381, status:SUCCESS 2023-05-15 14:12:31.664 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.665 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.665 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.665 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.666 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.666 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.667 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.667 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.667 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.668 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.668 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.672 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.672 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.673 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.676 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.678 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.678 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.678 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.685 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.685 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.686 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.693 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.694 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.694 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.694 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.694 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.694 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.695 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.695 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.695 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.695 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:12:31.695 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.696 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:12:31.706 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.707 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.707 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.707 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.707 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.708 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.708 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.708 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:12:31.709 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:12:31.718 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:12:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:12:31.734 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.734 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.735 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.735 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.735 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.737 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.737 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.737 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.739 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:31.740 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.741 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.741 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.741 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:31.747 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.747 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.748 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.753 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.755 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:12:31.755 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.756 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.756 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:12:31.762 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.765 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:12:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.766 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:31.766 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.767 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:12:31.767 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:12:31.767 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.767 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.768 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:12:31.777 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.777 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:12:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:31.780 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:12:31.780 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:12:31.780 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:12:31.784 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,fd82b0acdac7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131995, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:31.788 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:12:31.790 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:12:31.790 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:12:31.790 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:12:31.794 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a3488c552241] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131994, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:12:31.965 INFO 11 --- [pool-ocp-schedules-17,,be9aa8151157] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:31.965 INFO 11 --- [pool-ocp-schedules-17,,be9aa8151157] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:32.075 INFO 11 --- [pool-ocp-schedules-1,,da98a8540914] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:12:32.078 INFO 11 --- [pool-ocp-schedules-1,,da98a8540914] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:12:32.083 INFO 11 --- [pool-ocp-schedules-8,,f9eecb135c11] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:12:32.085 INFO 11 --- [pool-ocp-schedules-8,,f9eecb135c11] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:12:32.098 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.098 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.098 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.098 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.099 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.099 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.099 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.100 INFO 11 --- [ForkJoinPool-66078-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:12:32.101 INFO 11 --- [pool-ocp-schedules-18,,2c7de862570e] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:12:32.102 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.102 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.102 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.103 INFO 11 --- [pool-ocp-schedules-18,,2c7de862570e] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:12:32.103 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.107 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.107 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.108 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.111 INFO 11 --- [ForkJoinPool-66078-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:12:32.128 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.129 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.129 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.129 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.140 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.140 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.140 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:12:32.148 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.149 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.149 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.149 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.154 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.154 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.154 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:12:32.185 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:32.185 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.186 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.186 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.186 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.188 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.188 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.188 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:12:32.189 WARN 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:12:32.197 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.197 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.198 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:12:32.199 INFO 11 --- [ForkJoinPool-66080-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:12:32.199 INFO 11 --- [ForkJoinPool-66080-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:12:32.199 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:12:32.213 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.214 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:12:32.230 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:32.231 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.231 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.231 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.232 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.234 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.237 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.237 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.237 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:12:32.241 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.241 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.242 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:12:32.244 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:12:32.247 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:32.248 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.248 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.248 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.248 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.250 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:12:32.250 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.250 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.261 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:12:32.261 INFO 11 --- [ForkJoinPool-66080-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:12:32.262 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.262 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:12:32.280 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.280 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:12:32.295 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.295 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:12:32.311 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.311 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:12:32.328 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.328 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:12:32.342 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:12:32.358 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:32.359 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.359 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.359 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.360 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.364 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:32.364 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:32.365 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:32.387 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:12:32.387 INFO 11 --- [ForkJoinPool-66080-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:12:32.588 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,8ede2c47b51d4f20,d4f6bbcd79cd] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8ede2c47b51d4f20, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:32.593 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,8ede2c47b51d4f20,d4f6bbcd79cd] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8ede2c47b51d4f20, duration=6 ms] 2023-05-15 14:12:32.682 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116380, status:SUCCESS 2023-05-15 14:12:32.683 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116380, status:SUCCESS 2023-05-15 14:12:32.978 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:32.979 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:34.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:34.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:34.831091+08:00, lastProcessTime=2023-05-15T14:12:24.830125+08:00 2023-05-15 14:12:34.858 INFO 11 --- [alarm-detect-13,,b1846cf91503] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131134 ~ 1684131144 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.175, values={1684131140=246.175}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:34.865 INFO 11 --- [alarm-detect-11,,a48b2306f645] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131084 ~ 1684131144 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389517.938, values={1684131120=389517.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:12:34.866 INFO 11 --- [alarm-detect-11,,a48b2306f645] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131084 ~ 1684131144 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994315.107, values={1684131120=994315.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:34.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:34.831091+08:00 2023-05-15 14:12:35.678 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,ebe6c885c71145ae,c1b739e9f67f] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=ebe6c885c71145ae, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:35.683 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,ebe6c885c71145ae,c1b739e9f67f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=ebe6c885c71145ae, duration=5 ms] 2023-05-15 14:12:35.979 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:35.980 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:36.074 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,4fd0bd36febf4600,dbd4c58b94b3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=4fd0bd36febf4600, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:36.079 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,4fd0bd36febf4600,dbd4c58b94b3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=4fd0bd36febf4600, duration=5 ms] 2023-05-15 14:12:36.177 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:36.195 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:36.210 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:36.224 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:37.116 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:38.272 INFO 11 --- [pool-overview-container-5,,39cfa1df69c6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:38.522 INFO 11 --- [pool-scanner--15976825-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:38.980 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:38.981 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:39.092 INFO 11 --- [pool-ocp-schedules-5,,1eed58cd3a5c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:41.981 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:41.983 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:42.468 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:44.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:44.827156+08:00, lastProcessTime=2023-05-15T14:12:34.831091+08:00 2023-05-15 14:12:44.924 INFO 11 --- [alarm-detect-12,,171a46e3b599] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131144 ~ 1684131154 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.175, values={1684131150=246.175}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:44.926 INFO 11 --- [alarm-detect-12,,171a46e3b599] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:12:44.949 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:44.827156+08:00 2023-05-15 14:12:44.983 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:44.984 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:45.516 INFO 11 --- [pool-ocp-schedules-18,,b9f9e51adfa2] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:12:46.179 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:46.197 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:46.215 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:46.226 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:47.119 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:47.120 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118} 2023-05-15 14:12:47.145 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.145 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.146 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.146 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.147 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.148 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.153 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.153 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.154 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.154 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.158 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:47.059, end=2023-05-15T14:10:47.118},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:12:47.158 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:12:47.158 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:12:47.984 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:47.987 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:48.273 INFO 11 --- [pool-overview-container-3,,33546f3ef733] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:12:48.273 INFO 11 --- [pool-overview-container-3,,33546f3ef733] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:48.523 INFO 11 --- [pool-scanner--15976925-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:49.110 INFO 11 --- [pool-ocp-schedules-16,,194841f4b630] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:49.958 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,6504d58803ba45df,3dde14b30e87] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=6504d58803ba45df, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:12:49.959 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,6504d58803ba45df,3dde14b30e87] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=6504d58803ba45df, duration=2 ms] 2023-05-15 14:12:50.307 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sqls/E73CD9396760671FA7F09232821E6216/text?startTime=2023-05-15T13%3A40%3A44%2B08%3A00&endTime=2023-05-15T14%3A10%3A44%2B08%3A00, client=192.168.80.229, traceId=37450f6274c54664, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.sqlText(Long,Long,String,OffsetDateTime,OffsetDateTime), args=2,6,E73CD9396760671FA7F09232821E6216,2023-05-15T13:40:44+08:00,2023-05-15T14:10:44+08:00,] 2023-05-15 14:12:50.340 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.340 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.340 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:50.341 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:50.373 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.373 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:50.374 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:12:50.414 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:50.415 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.416 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:50.417 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:50.438 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:50.438 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:50.439 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT/*+ QUERY_TIMEOUT(30000000) */ con_id as tenant_id, object_id as database_id, object_name as database_name FROM oceanbase.cdb_objects where con_id = ? and OBJECT_TYPE = 'DATABASE' , args: [1002] 2023-05-15 14:12:50.544 INFO 11 --- [pool-ocp-schedules-8,,0c36c45f0baf] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:12:50.762 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,37450f6274c54664,22b50068b63c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/sqls/E73CD9396760671FA7F09232821E6216/text client=192.168.80.229, traceId=37450f6274c54664, duration=456 ms] 2023-05-15 14:12:50.987 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:50.989 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:51.860 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:51.872 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:51.875 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:51.971 INFO 11 --- [pool-ocp-schedules-9,,540b487086dd] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:51.971 INFO 11 --- [pool-ocp-schedules-9,,540b487086dd] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:12:52.190 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,bb6f00c05221457d,069c647b5fce] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=bb6f00c05221457d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:12:52.195 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,bb6f00c05221457d,069c647b5fce] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=bb6f00c05221457d, duration=5 ms] 2023-05-15 14:12:53.399 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5?id=2&tenantId=5, client=192.168.80.229, traceId=56f942342bad439b, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getTenant(Long,Long), args=2,5,] 2023-05-15 14:12:53.432 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.433 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.433 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.433 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:53.436 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,793d0c24e70c4083,09363952830c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/zones?id=2&tenantId=5, client=192.168.80.229, traceId=793d0c24e70c4083, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listObClusterZones(Long,Long), args=2,5,] 2023-05-15 14:12:53.467 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,793d0c24e70c4083,09363952830c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/zones client=192.168.80.229, traceId=793d0c24e70c4083, duration=32 ms] 2023-05-15 14:12:53.480 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:53.481 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.481 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.481 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.482 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,1948d44c5325] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2700,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40063} 2023-05-15 14:12:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,1948d44c5325] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:12:53.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.573 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.MAX_IOPS, t2.MIN_IOPS FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2023-05-15 14:12:53.583 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:53.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.585 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.585 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.586 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:53.601 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.601 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.602 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2023-05-15 14:12:53.677 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.677 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.678 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1] 2023-05-15 14:12:53.682 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.682 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.683 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:53.708 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.708 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.709 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:12:53.800 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:53.801 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:53.884 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.884 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.885 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:12:53.914 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=sys, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:12:53.914 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=sys, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:12:53.925 INFO 11 --- [obsdk-init-thread-4,56f942342bad439b,0f1af9881f4d] com.alibaba.druid.pool.DruidDataSource : {dataSource-103,ob-connector-datasource_proxy_obcluster_1_sys_root_127.0.0.1_2888} inited 2023-05-15 14:12:53.925 INFO 11 --- [obsdk-init-thread-4,56f942342bad439b,0f1af9881f4d] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=sys, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:12:53.926 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-05-15 14:12:53.926 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.935 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.935 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.936 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:53.982 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:53.982 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:53.983 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW GLOBAL VARIABLES 2023-05-15 14:12:53.989 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:53.990 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:54.054 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:54.054 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:54.055 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'TENANT' 2023-05-15 14:12:54.250 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:54.251 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:54.251 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:54.251 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:54.252 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:54.273 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:54.273 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:54.274 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2023-05-15 14:12:54.359 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,56f942342bad439b,cef13fdc6239] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5 client=192.168.80.229, traceId=56f942342bad439b, duration=962 ms] 2023-05-15 14:12:54.455 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,ebefb4cbd0124a9a,f6f1b6e89aef] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs/limitRule?obVersion=4.1.0.0, client=192.168.80.229, traceId=ebefb4cbd0124a9a, method=SuccessResponse com.oceanbase.ocp.server.controller.ObUnitSpecController.getUnitSpecLimitRule(String), args=4.1.0.0,] 2023-05-15 14:12:54.456 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,bb9fe7917be543ce,7e82d844ae3b] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs?null, client=192.168.80.229, traceId=bb9fe7917be543ce, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObUnitSpecController.listUnitSpecs(Pageable,String,List,String), args=Page request [number: 0, size 2000, sort: id: ASC],] 2023-05-15 14:12:54.459 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,ebefb4cbd0124a9a,f6f1b6e89aef] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs/limitRule client=192.168.80.229, traceId=ebefb4cbd0124a9a, duration=6 ms] 2023-05-15 14:12:54.461 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,bb9fe7917be543ce,7e82d844ae3b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs client=192.168.80.229, traceId=bb9fe7917be543ce, duration=8 ms] 2023-05-15 14:12:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:12:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:12:54.830614+08:00, lastProcessTime=2023-05-15T14:12:44.827156+08:00 2023-05-15 14:12:54.862 INFO 11 --- [alarm-detect-3,,ccc3c2c7ccbb] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131154 ~ 1684131164 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.22, values={1684131160=246.22}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:12:54.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:12:54.830614+08:00 2023-05-15 14:12:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,dac62bfcd078] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:12:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,4e591b1fa7be] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:12:55.496 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,3093ca8c439d4b8a,9e751b68f32f] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/topSql/switch?id=2&tenantId=5, client=192.168.80.229, traceId=3093ca8c439d4b8a, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitch(Long,Long), args=2,5,] 2023-05-15 14:12:55.497 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,5e7eae2d21324894,7850cd2de30d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/topSql/status/events?null, client=192.168.80.229, traceId=5e7eae2d21324894, method=PaginatedResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchEvents(Long,Long,OffsetDateTime,OffsetDateTime,Pageable), args=2,5,Page request [number: 0, size 2000, sort: timestamp: ASC],] 2023-05-15 14:12:55.500 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/performance/config?id=2&tenantId=5, client=192.168.80.229, traceId=f0f59777d51744c2, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getClusterPerfConfig(Long,Long), args=2,5,] 2023-05-15 14:12:55.513 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,5e7eae2d21324894,7850cd2de30d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/topSql/status/events client=192.168.80.229, traceId=5e7eae2d21324894, duration=20 ms] 2023-05-15 14:12:55.527 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,3093ca8c439d4b8a,9e751b68f32f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/topSql/switch client=192.168.80.229, traceId=3093ca8c439d4b8a, duration=34 ms] 2023-05-15 14:12:55.546 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/outlines?null, client=192.168.80.229, traceId=1658e403d9d74bfb, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObOutlineController.getSqlOutline(Long,Long,String,String,OffsetDateTime,OffsetDateTime,Boolean), args=2,5,] 2023-05-15 14:12:55.571 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.572 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:55.577 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.577 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2023-05-15 14:12:55.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() REGEXP 'OceanBase.*?CE' 2023-05-15 14:12:55.580 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.580 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.581 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2023-05-15 14:12:55.637 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8409a54b564e4e0b,e704c2c83e2c] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/topSql/status/switchOffEvents?startTime=2023-05-15T13%3A42%3A55%2B08%3A00&endTime=2023-05-15T14%3A12%3A55%2B08%3A00, client=192.168.80.229, traceId=8409a54b564e4e0b, method=SuccessResponse com.oceanbase.ocp.server.controller.perf.ObStatCollectionConfigController.queryTenantTopSqlSwitchOffIntervalLimit3(Long,Long,OffsetDateTime,OffsetDateTime), args=2,5,2023-05-15T13:42:55+08:00,2023-05-15T14:12:55+08:00,] 2023-05-15 14:12:55.645 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/topSql?startTime=2023-05-15T13%3A42%3A55%2B08%3A00&endTime=2023-05-15T14%3A12%3A55%2B08%3A00&inner=false, client=192.168.80.229, traceId=d73c8d61a1844f09, method=IterableResponse com.oceanbase.ocp.server.controller.perf.ObSqlStatController.topSql(Long,Long,OffsetDateTime,OffsetDateTime,Long,Boolean,String,String,String,String,Long,RequestingSqlText,Integer,List,String), args=2,5,2023-05-15T13:42:55+08:00,2023-05-15T14:12:55+08:00,false,] 2023-05-15 14:12:55.645 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:12:55.646 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:12:55.656 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,8409a54b564e4e0b,e704c2c83e2c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/topSql/status/switchOffEvents client=192.168.80.229, traceId=8409a54b564e4e0b, duration=20 ms] 2023-05-15 14:12:55.674 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.675 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.675 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.676 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:55.684 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.684 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.685 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:12:55.689 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Query sql audit with param:QueryTopSqlParam(interval=Interval{start=2023-05-15T13:42:55, end=2023-05-15T14:12:55}, clusterId=2, tenantId=5, serverId=null, inner=false, sqlText=null, search=null, limit=null, requestingSqlText=ASSUME_HITS, sqlTextLength=null, customColumns=null, filterExpression=null, parseSqlType=false, parseStatement=false, parseTable=false, groupByServer=true) 2023-05-15 14:12:55.693 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : split query range to:{0=Interval{start=2023-05-15T13:42:55, end=2023-05-15T14:12:55}} 2023-05-15 14:12:55.713 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.713 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.713 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2023-05-15 14:12:55.739 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlAuditRawStatServiceImpl : Get sql info from cache and monitor db 2023-05-15 14:12:55.746 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:12:55.749 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.749 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.750 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2023-05-15 14:12:55.771 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.772 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.772 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.773 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:12:55.778 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.778 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.779 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:12:55.787 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,1658e403d9d74bfb,ebe3e7b875b4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/outlines client=192.168.80.229, traceId=1658e403d9d74bfb, duration=242 ms] 2023-05-15 14:12:55.800 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check filter expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:12:55.800 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.p.s.i.SqlDiagnoseExpressionChecker : check extra column expressions:null, with :SqlAuditStatSummary(super=SqlAuditStatBase(executions=101120.0, execPs=56.62, avgAffectedRows=1.0, avgReturnRows=0.0, avgPartitionCount=1.0, failCount=0, failPercentage=0.0, retCode4012Count=0, retCode4013Count=0, retCode5001Count=0, retCode5024Count=0, retCode5167Count=0, retCode5217Count=0, retCode6002Count=0, avgWaitTime=0.0, sumWaitTime=73.53, avgWaitCount=0.0, avgRpcCount=0.01, localPlanPercentage=100.0, remotePlanPercentage=0.0, remotePlans=0, distPlanPercentage=0.0, avgElapsedTime=0.77, maxElapsedTime=56.75, sumElapsedTime=77523.69, avgCpuTime=0.71, maxCpuTime=56.69, avgNetTime=0.0, avgNetWaitTime=0.0, avgQueueTime=0.02, avgDecodeTime=0.0, avgGetPlanTime=0.1, avgExecuteTime=0.62, avgExecutorRpcCount=0.0, missPlanPercentage=0.0, missPlans=0, avgApplicationWaitTime=0.0, avgConcurrencyWaitTime=0.0, avgUserIoWaitTime=0.0, avgScheduleTime=0.0, avgRowCacheHit=0.0, avgBloomFilterCacheHit=1.83, avgBlockCacheHit=1.49, avgBlockIndexCacheHit=0.0, avgDiskReads=0.0, retryCount=0, tableScanPercentage=0.0, strongConsistencyPercentage=100.0, weakConsistencyPercentage=0.0, avgMemstoreReadRows=2.0, avgSsstoreReadRows=2.25, maxAffectedRows=null, maxReturnRows=null, maxWaitTime=null, maxApplicationWaitTime=null, maxConcurrencyWaitTime=null, maxUserIoWaitTime=null, maxDiskReads=null, avgExpectedWorkerCount=null, avgUsedWorkerCount=null, avgLogicalReads=5.15, sumLogicalReads=520768), sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, server=10.xxx.xx.12:2982, serverIp=10.xxx.xx.12, serverPort=null, dbName=NO_database, userName=root, sqlType=UPDATE, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , inner=false, waitEvent=db file data read, customColumns=null, startTimeUs=1683880862000000, endTimeUs=1683882648000000, obServerId=7, obUserId=200001, obDbId=1101710651531551, sqlInfo=SqlInfo(clusterId=1, clusterName=obcluster, tenantId=1002, serverId=1, server=10.xxx.xx.11:2982, dbId=1101710651531551, dbName=NO_database, userId=200001, userName=root, sqlId=63577C163A26A8D3FC2EA6A2C92C4E83, sqlText=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = '20230512', `order_status` = '1', `suc_order_time` = '20230512075941', `fee` = '0', `netr_amt` = '500', `isaddaccountflag` = '1', `discount_fee` = '1', `discount_flag` = '1', `t_order_no` = 'A0230512075935396992', `re_bank_code` = 'OTHERS', `re_txamt` = '500', `t_pay_no` = '4200001876202305126249546444', `acc_amt` = '194000', `rate` = '0.000000', `act_id` = '63643', `gfshcz_money` = '0' WHERE `order_no` = '20230512TT1q5c61' and `order_time` = '20230512075935' and `cust_id` = '60000003190816', statement=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, `order_status` = ?, `suc_order_time` = ?, `fee` = ?, `netr_amt` = ?, `isaddaccountflag` = ?, `discount_fee` = ?, `discount_flag` = ?, `t_order_no` = ?, `re_bank_code` = ?, `re_txamt` = ?, `t_pay_no` = ?, `acc_amt` = ?, `rate` = ?, `act_id` = ?, `gfshcz_money` = ? WHERE `order_no` = ? and `order_time` = ? and `cust_id` = ?, sqlTextShort=UPDATE `toob`.`yyf_tcodepay_t` SET `ustldat` = ?, , sqlType=UPDATE, tableList=[SqlStatUtils.Table(dbName=toob, tableName=yyf_tcodepay_t)], createTime=2023-05-12 08:05:21.0, sensitive=false, mode=MYSQL)) 2023-05-15 14:12:55.805 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,d73c8d61a1844f09,ebfbd2070641] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/topSql client=192.168.80.229, traceId=d73c8d61a1844f09, duration=162 ms] 2023-05-15 14:12:55.942 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:55.942 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:55.943 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:12:56.182 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:56.199 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:56.217 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:56.228 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:56.248 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,f0f59777d51744c2,50744a53a5b3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/performance/config client=192.168.80.229, traceId=f0f59777d51744c2, duration=749 ms] 2023-05-15 14:12:56.990 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:56.991 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:12:57.124 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:57.502 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:12:57.804 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c0531a32083345c7,a5f08bfeefe5] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/compaction?id=2&tenantId=5, client=192.168.80.229, traceId=c0531a32083345c7, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantCompactionController.getTenantCompaction(Long,Long), args=2,5,] 2023-05-15 14:12:57.833 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c0531a32083345c7,a5f08bfeefe5] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:57.833 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c0531a32083345c7,a5f08bfeefe5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:57.834 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c0531a32083345c7,a5f08bfeefe5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM DBA_OB_MAJOR_COMPACTION 2023-05-15 14:12:57.878 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c0531a32083345c7,a5f08bfeefe5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/compaction client=192.168.80.229, traceId=c0531a32083345c7, duration=75 ms] 2023-05-15 14:12:57.923 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,84ec6b16f7ea4366,ca132269fbf5] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metric?startTime=2023-05-11T02%3A00%3A02%2B08%3A00&endTime=2023-05-15T14%3A12%3A57%2B08%3A00&metrics=memstore_percent%2Ctotal_memstore_used&labels=app%3AOB%2Cobregion%3Aobcluster%2Cob_cluster_id%3A1%2Ctenant_name%3Asys&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&maxPoints=360, client=192.168.80.229, traceId=84ec6b16f7ea4366, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetric(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Long,Integer,String), args=2023-05-11T02:00:02+08:00,2023-05-15T14:12:57+08:00,memstore_percent,total_memstore_used,app,obregion,ob_cluster_id,tenant_name,0,0,360,app:OB,obregion:obcluster,ob_cluster_id:1,tenant_name:sys,] 2023-05-15 14:12:58.015 INFO 11 --- [pool-scanner--15977019-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=327, nodesFilledCount=113 2023-05-15 14:12:58.015 INFO 11 --- [pool-scanner--15977020-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=327, nodesFilledCount=113 2023-05-15 14:12:58.030 INFO 11 --- [pool-scanner--15977021-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=327, nodesFilledCount=113 2023-05-15 14:12:58.034 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,84ec6b16f7ea4366,ca132269fbf5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metric client=192.168.80.229, traceId=84ec6b16f7ea4366, duration=113 ms] 2023-05-15 14:12:58.273 INFO 11 --- [pool-overview-container-7,,012b3289dbe1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:12:58.520 INFO 11 --- [pool-scanner--15977022-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:12:59.132 INFO 11 --- [pool-ocp-schedules-11,,b650bdd4af73] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:12:59.241 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/sessions?page=1&size=10&activeOnly=false, client=192.168.80.229, traceId=c4a72b4c48874ec1, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.listTenantSessions(Pageable,Long,Long,String,String,String,Boolean), args=Page request [number: 0, size 10, sort: id: DESC],2,5,false,] 2023-05-15 14:12:59.266 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:59.266 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:59.267 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2023-05-15 14:12:59.301 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:12:59.301 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:12:59.302 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? ORDER BY id DESC LIMIT 0,10, args: [sys] 2023-05-15 14:12:59.370 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,c4a72b4c48874ec1,8abb80be9dfe] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/sessions client=192.168.80.229, traceId=c4a72b4c48874ec1, duration=132 ms] 2023-05-15 14:12:59.991 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:12:59.992 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:13:00.000 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:13:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:13:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131180 2023-05-15 14:13:00.050 INFO 11 --- [pool-ocp-schedules-13,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:13:00.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:13:00.0, currentDate=Mon May 15 14:13:00 CST 2023 2023-05-15 14:13:00.666 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:13:00.0} 2023-05-15 14:13:00.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:13:00.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:00.671 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:00.679 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:13:00.0, currentDate=Mon May 15 14:13:00 CST 2023 2023-05-15 14:13:00.679 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:13:00.0} 2023-05-15 14:13:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:13:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:00.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:13:00.0, currentDate=Mon May 15 14:13:00 CST 2023 2023-05-15 14:13:00.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:13:00.0} 2023-05-15 14:13:00.706 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:13:00.707 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:00.707 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:00.711 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:31.0, nextRun=2023-05-15 14:13:00.0, currentDate=Mon May 15 14:13:00 CST 2023 2023-05-15 14:13:00.712 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:12:31.0, nextRunTime=2023-05-15 14:13:00.0} 2023-05-15 14:13:00.715 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:13:00.715 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:00.715 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:13:00.0, currentDate=Mon May 15 14:13:00 CST 2023 2023-05-15 14:13:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:13:00.0} 2023-05-15 14:13:00.726 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:13:00.726 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:00.726 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:00.873 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132000, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:00.889 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132000, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:13:00.870+08:00, endTime=null} 2023-05-15 14:13:00.908 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,19fe12aea230] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132000, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116385, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:00.863+08:00, sub_task_instance_id=132000}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:00.908 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,19fe12aea230] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:13:00.915 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132001, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:00.928 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:13:00.928 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:13:00.928 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:13:00.928 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:13:00.930 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:13:00.932 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:13:00.933 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132001, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:13:00.899+08:00, endTime=null} 2023-05-15 14:13:00.941 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.941 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.941 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.942 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.942 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.943 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.943 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.943 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.944 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.944 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.944 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:13:00.944 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.944 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.945 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.946 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.946 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.947 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:13:00.949 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:13:00.949 INFO 11 --- [pool-3247239-thread-4,fb27d19334114ed7,d8f6677f1aff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:13:00.949 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131998, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:00.949 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.955 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.955 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.955 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:13:00.956 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:13:00.956 INFO 11 --- [pool-3247239-thread-3,fb27d19334114ed7,54ff16bc4356] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:13:00.963 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:13:00.963 INFO 11 --- [pool-3247239-thread-1,fb27d19334114ed7,9577cfd1f990] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:13:00.965 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.966 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.966 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.966 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.967 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:00.967 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.970 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:13:00.974 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.974 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:13:00.974 INFO 11 --- [pool-3247239-thread-2,fb27d19334114ed7,6d4980a25190] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:13:00.976 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.976 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.977 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.977 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131998, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:13:00.946+08:00, endTime=null} 2023-05-15 14:13:00.978 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01590c7c5cb2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132001, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116386, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:00.899+08:00, sub_task_instance_id=132001}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:00.980 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.983 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.984 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.984 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.984 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.984 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:13:00.985 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:00.989 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:00.990 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:00.990 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:13:00.990 INFO 11 --- [pool-3247239-thread-6,fb27d19334114ed7,44d4128d58f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:13:00.990 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:13:00.996 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:13:00.996 INFO 11 --- [pool-3247239-thread-5,fb27d19334114ed7,8ff624e2e556] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:13:00.997 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,19fe12aea230] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:13:00.998 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01590c7c5cb2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132001, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:00.999 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131999, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:01.001 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c4b62af0719d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131998, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116383, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:00.945+08:00, sub_task_instance_id=131998}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:01.004 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131999, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:13:00.998+08:00, endTime=null} 2023-05-15 14:13:01.004 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,19fe12aea230] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132000, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:01.025 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c4b62af0719d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:13:01.026 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c4b62af0719d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:13:01.027 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131999, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116384, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:00.997+08:00, sub_task_instance_id=131999}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:01.030 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c4b62af0719d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131998, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:01.039 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131997, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:01.047 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:13:01.047 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:13:01.050 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:13:01.050 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:13:01.058 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=131997, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:13:01.017+08:00, endTime=null} 2023-05-15 14:13:01.096 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,2d21159f66eb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131999, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:01.123 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d1012fda400e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=131997, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116382, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:01.015+08:00, sub_task_instance_id=131997}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:01.129 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:13:01.130 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:13:01.131 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:13:01.131 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:13:01.136 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:13:01.136 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,d1012fda400e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 131997, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:01.138 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:13:01.139 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:13:01.141 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:13:01.141 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:13:01.144 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:13:01.163 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:13:01.165 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:13:01.166 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:13:01.166 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:13:01.180 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:13:01.266 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116382, status:SUCCESS 2023-05-15 14:13:01.268 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116382, status:SUCCESS 2023-05-15 14:13:01.279 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116383, status:SUCCESS 2023-05-15 14:13:01.280 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116383, status:SUCCESS 2023-05-15 14:13:01.289 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116384, status:SUCCESS 2023-05-15 14:13:01.290 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116384, status:SUCCESS 2023-05-15 14:13:01.300 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116385, status:SUCCESS 2023-05-15 14:13:01.301 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116385, status:SUCCESS 2023-05-15 14:13:01.309 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116386, status:SUCCESS 2023-05-15 14:13:01.310 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116386, status:SUCCESS 2023-05-15 14:13:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,4dd15c444ddc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131240, interval=60 2023-05-15 14:13:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,c540448a44ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131240, interval=60 2023-05-15 14:13:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,c540448a44ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:13:01.523 INFO 11 --- [pool-prometheus-minute-schedule-1,,c540448a44ad] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131181, count=6, elapsed=1682447050342 2023-05-15 14:13:01.523 INFO 11 --- [pool-prometheus-minute-schedule-2,,4dd15c444ddc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:13:01.524 INFO 11 --- [pool-prometheus-minute-schedule-2,,4dd15c444ddc] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131181, count=8, elapsed=1682447050343 2023-05-15 14:13:02.992 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:02.994 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:04.204 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/5/parameters?null, client=192.168.80.229, traceId=9532a71539e84357, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantParameterController.listTenantParameters(Long,Long,Boolean,Boolean), args=2,5,] 2023-05-15 14:13:04.231 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:04.231 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:04.232 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW GLOBAL VARIABLES 2023-05-15 14:13:04.249 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:04.249 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:04.250 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'TENANT' 2023-05-15 14:13:04.353 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,9532a71539e84357,9b503a5a0776] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/5/parameters client=192.168.80.229, traceId=9532a71539e84357, duration=150 ms] 2023-05-15 14:13:04.714 INFO 11 --- [pool-ocp-schedules-12,,ca4e60a08117] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:13:04.721 INFO 11 --- [pool-ocp-schedules-12,,ca4e60a08117] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:13:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:04.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:04.830411+08:00, lastProcessTime=2023-05-15T14:12:54.830614+08:00 2023-05-15 14:13:04.853 INFO 11 --- [alarm-detect-9,,4faae1366265] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131164 ~ 1684131174 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.265, values={1684131170=246.265}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:04.885 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:04.830411+08:00 2023-05-15 14:13:05.427 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,4778f1cd52b447a1,f09b5ecbbc6e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=4778f1cd52b447a1, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:05.433 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,4778f1cd52b447a1,f09b5ecbbc6e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=4778f1cd52b447a1, duration=7 ms] 2023-05-15 14:13:05.871 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:06.0, nextRun=2023-05-15 14:13:05.0, currentDate=Mon May 15 14:13:05 CST 2023 2023-05-15 14:13:05.871 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:12:06.0, nextRunTime=2023-05-15 14:13:05.0} 2023-05-15 14:13:05.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:13:05.874 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:05.875 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:05.994 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:05.995 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:06.172 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132002, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:06.174 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132002, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:13:06.167+08:00, endTime=null} 2023-05-15 14:13:06.184 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:06.198 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c85ed901b2bd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132002, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116387, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:06.167+08:00, sub_task_instance_id=132002}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:06.201 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:06.219 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:06.223 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,55fd7f952a8f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:13:06.223 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,be23ed56b575] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:13:06.223 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,53fcbe84b62f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:13:06.234 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:06.264 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,be23ed56b575] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:13:06.265 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,53fcbe84b62f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:13:06.266 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,55fd7f952a8f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:13:06.279 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c85ed901b2bd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132002, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:06.415 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116387, status:SUCCESS 2023-05-15 14:13:06.416 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116387, status:SUCCESS 2023-05-15 14:13:06.495 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,7eb3585ca88e4086,522e64eb4020] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=7eb3585ca88e4086, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:06.500 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,7eb3585ca88e4086,522e64eb4020] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=7eb3585ca88e4086, duration=6 ms] 2023-05-15 14:13:07.127 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:08.273 INFO 11 --- [pool-overview-container-1,,822c7f439149] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:08.520 INFO 11 --- [pool-scanner--15977115-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:13:08.995 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:08.996 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:09.155 INFO 11 --- [pool-ocp-schedules-19,,efa346602b75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:09.950 INFO 11 --- [pool-ocp-schedules-6,,d1c717da1aed] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5390,"seriesIdKeySize":53532,"secondWriteQueue":0,"minuteWriteCacheSize":40066} 2023-05-15 14:13:09.950 INFO 11 --- [pool-ocp-schedules-6,,d1c717da1aed] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:13:11.688 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131120, nodeCount=61, affectRows=61 2023-05-15 14:13:11.689 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131120, nodeCount=61, validCount=61 2023-05-15 14:13:11.744 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131120, nodeCount=1000, affectRows=1000 2023-05-15 14:13:11.753 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131120, nodeCount=1000, validCount=1000 2023-05-15 14:13:11.797 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131120, nodeCount=1000, affectRows=1000 2023-05-15 14:13:11.805 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131120, nodeCount=1000, validCount=1000 2023-05-15 14:13:11.848 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131120, nodeCount=639, affectRows=639 2023-05-15 14:13:11.855 INFO 11 --- [pool-metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131120, nodeCount=639, validCount=639 2023-05-15 14:13:11.976 INFO 11 --- [pool-ocp-schedules-6,,1374c06a398a] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:11.977 INFO 11 --- [pool-ocp-schedules-6,,1374c06a398a] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:11.996 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:11.997 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:12.504 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:14.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:14.830540+08:00, lastProcessTime=2023-05-15T14:13:04.830411+08:00 2023-05-15 14:13:14.869 INFO 11 --- [alarm-detect-7,,675a5570830f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131174 ~ 1684131184 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.265, values={1684131180=246.265}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:14.949 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:14.830540+08:00 2023-05-15 14:13:14.998 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:14.999 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:16.186 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:16.203 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:16.221 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:16.236 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:16.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,9fbd33729c194955,e18919e9f694] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=9fbd33729c194955, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:16.807 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,9fbd33729c194955,e18919e9f694] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=9fbd33729c194955, duration=6 ms] 2023-05-15 14:13:17.130 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:13:17.628 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:17.628 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.629 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:17.631 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:17.631 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.631 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:13:17.651 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:17.652 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:17.653 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:17.653 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:13:17.663 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:13:17.663 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:13:17.663 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:13:17.663 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:13:17.669 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:13:17.669 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.669 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:13:17.670 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.670 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:13:17.670 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:13:17.670 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:17.671 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:13:17.671 INFO 11 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:13:17.693 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:10:48, start=2023-05-15T14:08:17, end=2023-05-15T14:10:47, updates=95, granule=120000ms, delay=151000ms 2023-05-15 14:13:17.693 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:11:02, start=2023-05-15T14:08:17, end=2023-05-15T14:10:47, updates=101, granule=120000ms, delay=165000ms 2023-05-15 14:13:17.695 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:10:48, target=2023-05-15T14:10:47, granule=120000ms, delay=1000ms) 2023-05-15 14:13:17.695 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:03:17, granule=600000ms, delay=450000ms) 2023-05-15 14:13:17.695 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:11:02, target=2023-05-15T14:10:47, granule=120000ms, delay=15000ms) 2023-05-15 14:13:17.695 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:05:17, granule=600000ms, delay=330000ms) 2023-05-15 14:13:17.699 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:10:48, start=2023-05-15T14:08:17, end=2023-05-15T14:10:47, updates=127, granule=120000ms, delay=151000ms 2023-05-15 14:13:17.701 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:10:48, target=2023-05-15T14:10:47, granule=120000ms, delay=1000ms) 2023-05-15 14:13:17.701 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:03:17, granule=600000ms, delay=450000ms) 2023-05-15 14:13:17.701 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=114ms, period=120000ms 2023-05-15 14:13:17.999 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:18.000 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:18.272 INFO 11 --- [pool-overview-container-6,,b3b23b68c2c6] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:18.519 INFO 11 --- [pool-scanner--15977215-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:19.177 INFO 11 --- [pool-ocp-schedules-7,,2630cae9f1c1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:20.363 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:20.0, nextRun=2023-05-15 14:13:20.0, currentDate=Mon May 15 14:13:20 CST 2023 2023-05-15 14:13:20.363 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:12:20.0, nextRunTime=2023-05-15 14:13:20.0} 2023-05-15 14:13:20.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:13:20.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:20.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:21.000 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:21.001 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:21.391 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132003, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:21.393 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132003, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:13:21.389+08:00, endTime=null} 2023-05-15 14:13:21.413 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,275a6c599956] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132003, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116388, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:21.389+08:00, sub_task_instance_id=132003}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:21.415 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,275a6c599956] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:13:21.415 INFO 11 --- [ForkJoinPool-66083-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:13:21.415 INFO 11 --- [ForkJoinPool-66083-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:13:21.419 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,275a6c599956] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132003, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:21.827 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116388, status:SUCCESS 2023-05-15 14:13:21.829 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116388, status:SUCCESS 2023-05-15 14:13:21.862 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:21.902 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:21.915 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:22.436 INFO 11 --- [pool-ocp-schedules-9,,4cb3923687d3] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:13:22.436 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:13:22.436 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:13:22.436 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:13:22.436 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:13:22.436 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:13:22.443 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:13:22.443 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:13:22.443 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:13:22.443 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:13:22.443 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:13:22.467 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:13:22.468 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:13:22.469 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:13:22.470 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:13:22.469887608+08:00 2023-05-15 14:13:22.470 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:13:22.469854714+08:00 2023-05-15 14:13:22.470 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:13:22.470+08:00 2023-05-15 14:13:22.470 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:13:22.470+08:00 2023-05-15 14:13:22.472 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:13:22.471523734+08:00 2023-05-15 14:13:22.472 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:13:22.472 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:13:22.472+08:00 2023-05-15 14:13:22.473 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:13:22.470+08:00 and status AVAILABLE 2023-05-15 14:13:22.473 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:13:22.470+08:00 and status AVAILABLE 2023-05-15 14:13:22.473 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:13:22.474 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:13:22.473661089+08:00 2023-05-15 14:13:22.474 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:13:22.474+08:00 2023-05-15 14:13:22.475 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:13:22.472+08:00 and status AVAILABLE 2023-05-15 14:13:22.476 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:13:22.475693214+08:00 2023-05-15 14:13:22.476 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:13:22.476+08:00 2023-05-15 14:13:22.478 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:13:22.474+08:00 and status AVAILABLE 2023-05-15 14:13:22.479 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:13:22.480 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:13:22.480 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:13:22.476+08:00 and status AVAILABLE 2023-05-15 14:13:22.481 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:13:22.495 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131202 0 0 , error= 2023-05-15 14:13:22.499 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131202 0 0 , error= 2023-05-15 14:13:22.509 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:13:22.510 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131202 0 0 , error= 2023-05-15 14:13:22.510 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:13:22.508+08:00 2023-05-15 14:13:22.511 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:13:22.511+08:00 2023-05-15 14:13:22.511 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:13:22.511+08:00 2023-05-15 14:13:22.525 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131202 0 0 , error= 2023-05-15 14:13:22.526 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:13:22.525+08:00 2023-05-15 14:13:22.526 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:13:22.508+08:00 and status AVAILABLE 2023-05-15 14:13:22.526 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:13:22.511+08:00 and status AVAILABLE 2023-05-15 14:13:22.526 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:13:22.511+08:00 and status AVAILABLE 2023-05-15 14:13:22.527 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:13:22.529 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:13:22.529 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:13:22.530 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:13:22.541 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131202 0 0 , error= 2023-05-15 14:13:22.544 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:13:22.543+08:00 2023-05-15 14:13:22.544 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:13:22.525+08:00 and status AVAILABLE 2023-05-15 14:13:22.547 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:13:22.547 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:13:22.543+08:00 and status AVAILABLE 2023-05-15 14:13:22.550 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:13:22.554 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:13:22.556 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:13:22.556 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:13:22.560 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:13:22.562 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:13:22.585 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:13:22.585 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:13:22.588 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:13:22.594 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:13:22.595 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:13:22.630 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:13:22.641 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:13:22.643 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:13:22.648 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:13:22.652 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:13:22.656 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:13:22.658 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:13:22.666 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:13:22.669 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:13:22.674 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:13:22.675 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:13:22.677 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:13:22.686 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:13:22.687 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:13:22.702 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:13:22.711 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:13:22.724 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:13:22.725 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:13:22.732 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:13:22.736 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:13:22.743 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:13:22.744 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:13:22.745 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:13:22.755 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:13:22.758 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:13:22.764 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:13:22.777 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:13:22.791 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:13:22.796 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:13:22.806 INFO 11 --- [pool-Check host status1,,82ba5d682061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:13:22.810 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:13:22.811 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:13:22.825 INFO 11 --- [pool-Check host status14,,8603be4fa477] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:13:22.828 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:13:22.829 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:13:22.836 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:13:22.840 INFO 11 --- [pool-Check host status8,,ef6e3cff60be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:13:22.848 INFO 11 --- [pool-Check host status30,,5143fda54115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:13:22.867 INFO 11 --- [pool-scanner--15977216-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:22.884 INFO 11 --- [pool-scanner--15977217-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:22.903 INFO 11 --- [pool-scanner--15977218-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:22.917 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:13:22.921 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:22.926 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:22.929 INFO 11 --- [pool-Check host status13,,64a19d979d05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:13:22.951 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:13:22.951 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:13:22.951 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:13:22.951 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:13:22.951 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:13:22.956 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:13:22.957 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:13:22.957 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:13:22.957 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:13:22.961 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:13:22.967 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:22.971 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:22.981 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:13:22.982 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:13:22.982 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:13:22.984 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:13:22.986 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:22.988 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:13:22.989 INFO 11 --- [pool-ocp-schedules-5,,f8bbe0a931d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status3,,cdc62a4f4595] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:13:23.002 INFO 11 --- [pool-Check host status17,,9538fad32326] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:13:23.004 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:13:23.004 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:13:23.004 INFO 11 --- [pool-Check host status19,,4eca530c5f2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:13:23.005 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:13:23.006 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:13:23.006 INFO 11 --- [pool-Check host status28,,d64102a877c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:13:23.011 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:13:23.011 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:13:23.011 INFO 11 --- [pool-Check host status2,,078f39ff5ac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:13:23.446 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ae4a7900fc234217,20708ea63c1e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ae4a7900fc234217, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:23.451 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ae4a7900fc234217,20708ea63c1e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ae4a7900fc234217, duration=6 ms] 2023-05-15 14:13:23.995 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,648ca4ef806546ee,d1e73776f7b6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=648ca4ef806546ee, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:13:23.997 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,648ca4ef806546ee,d1e73776f7b6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=648ca4ef806546ee, duration=2 ms] 2023-05-15 14:13:24.001 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:24.002 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:24.831241+08:00, lastProcessTime=2023-05-15T14:13:14.830540+08:00 2023-05-15 14:13:24.869 INFO 11 --- [alarm-detect-12,,7298db78cd70] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131184 ~ 1684131194 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.265, values={1684131190=246.265}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:24.871 INFO 11 --- [alarm-detect-14,,ecd5a10feb2c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131134 ~ 1684131194 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389577.988, values={1684131180=389577.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:13:24.873 INFO 11 --- [alarm-detect-14,,ecd5a10feb2c] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131134 ~ 1684131194 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994375.358, values={1684131180=994375.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:24.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:24.831241+08:00 2023-05-15 14:13:25.526 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:26.0, nextRun=2023-05-15 14:13:25.0, currentDate=Mon May 15 14:13:25 CST 2023 2023-05-15 14:13:25.526 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:12:26.0, nextRunTime=2023-05-15 14:13:25.0} 2023-05-15 14:13:25.536 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:13:25.536 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:25.536 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:26.087 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,46b4a7f5839d4b10,fae3f588e84b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=46b4a7f5839d4b10, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:26.092 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,46b4a7f5839d4b10,fae3f588e84b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=46b4a7f5839d4b10, duration=5 ms] 2023-05-15 14:13:26.189 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:26.206 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:26.223 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:26.238 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:26.477 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132004, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:26.479 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132004, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:13:26.473+08:00, endTime=null} 2023-05-15 14:13:26.500 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132004, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116389, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:26.473+08:00, sub_task_instance_id=132004}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:13:26.577 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:13:26.578 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:13:26.579 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:13:26.579 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:13:26.580 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:13:26.584 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4178d7777757] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132004, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:26.943 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116389, status:SUCCESS 2023-05-15 14:13:26.945 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116389, status:SUCCESS 2023-05-15 14:13:27.002 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:27.003 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:27.133 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:27.506 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:28.272 INFO 11 --- [pool-overview-container-2,,228c0ded4783] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:28.519 INFO 11 --- [pool-scanner--15977316-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:29.198 INFO 11 --- [pool-ocp-schedules-14,,105ca9c38e71] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:30.004 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:30.005 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:30.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:31.0, nextRun=2023-05-15 14:13:30.0, currentDate=Mon May 15 14:13:30 CST 2023 2023-05-15 14:13:30.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:12:31.0, nextRunTime=2023-05-15 14:13:30.0} 2023-05-15 14:13:30.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:13:30.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:30.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:30.693 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:30.701 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:01.0, nextRun=2023-05-15 14:13:30.0, currentDate=Mon May 15 14:13:30 CST 2023 2023-05-15 14:13:30.702 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:13:01.0, nextRunTime=2023-05-15 14:13:30.0} 2023-05-15 14:13:30.705 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:13:30.705 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:13:30.705 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:13:31.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,abf4ec06ec634549,1e4b3a54ca77] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=192.168.80.229, traceId=abf4ec06ec634549, method=SuccessResponse com.oceanbase.ocp.server.controller.ProfileController.userInfo(), args=] 2023-05-15 14:13:31.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,ca64809db68340f4,17a654fa29df] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.80.229, traceId=ca64809db68340f4, method=OffsetDateTime com.oceanbase.ocp.server.controller.InfoController.time(), args=] 2023-05-15 14:13:31.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,ca64809db68340f4,17a654fa29df] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.80.229, traceId=ca64809db68340f4, duration=0 ms] 2023-05-15 14:13:31.448 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,227d4c7d949841b3,5d3c934ba400] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/maz/state?null, client=192.168.80.229, traceId=227d4c7d949841b3, method=SuccessResponse com.oceanbase.ocp.server.controller.MultiAzoneController.state(), args=] 2023-05-15 14:13:31.450 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,227d4c7d949841b3,5d3c934ba400] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/maz/state client=192.168.80.229, traceId=227d4c7d949841b3, duration=2 ms] 2023-05-15 14:13:31.451 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,d10ff8eae2dc4588,8890d930fa46] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=192.168.80.229, traceId=d10ff8eae2dc4588, method=Map com.oceanbase.ocp.server.controller.InfoController.info(), args=] 2023-05-15 14:13:31.451 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,d10ff8eae2dc4588,8890d930fa46] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=192.168.80.229, traceId=d10ff8eae2dc4588, duration=0 ms] 2023-05-15 14:13:31.456 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,abf4ec06ec634549,1e4b3a54ca77] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=192.168.80.229, traceId=abf4ec06ec634549, duration=9 ms] 2023-05-15 14:13:31.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,e21545c8764d4530,efa692b2dd6b] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=192.168.80.229, traceId=e21545c8764d4530, method=SuccessResponse com.oceanbase.ocp.server.controller.PropertyController.getSystemInfo(), args=] 2023-05-15 14:13:31.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,b71e0567f7e14d5f,2964d34d67d7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/mc/info?null, client=192.168.80.229, traceId=b71e0567f7e14d5f, method=SuccessResponse com.oceanbase.ocp.server.controller.OcpMultiClusterInfoController.info(), args=] 2023-05-15 14:13:31.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,e21545c8764d4530,efa692b2dd6b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=192.168.80.229, traceId=e21545c8764d4530, duration=0 ms] 2023-05-15 14:13:31.463 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,b71e0567f7e14d5f,2964d34d67d7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/mc/info client=192.168.80.229, traceId=b71e0567f7e14d5f, duration=0 ms] 2023-05-15 14:13:31.464 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,cbba4f73896a4b6b,ed976d415922] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/loginKey?null, client=192.168.80.229, traceId=cbba4f73896a4b6b, method=SuccessResponse com.oceanbase.ocp.server.controller.InfoController.getLoginKey(), args=] 2023-05-15 14:13:31.464 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,cbba4f73896a4b6b,ed976d415922] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/loginKey client=192.168.80.229, traceId=cbba4f73896a4b6b, duration=1 ms] 2023-05-15 14:13:31.487 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,5d66923db4b64660,29f6ea8446d5] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/iam/user/listAll?null, client=192.168.80.229, traceId=5d66923db4b64660, method=IterableResponse com.oceanbase.ocp.server.controller.IamController.listAllUserInfo(), args=] 2023-05-15 14:13:31.488 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,3f8024e0c7364fb4,c845a6be3e2a] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=OB&keyword=, client=192.168.80.229, traceId=3f8024e0c7364fb4, method=PaginatedResponse com.oceanbase.ocp.server.controller.ProfileController.listCredentials(AccessTargetType,String,Pageable), args=OB,,Page request [number: 0, size 2000, sort: create_time: DESC],] 2023-05-15 14:13:31.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,3941237c36724298,160b91699fdb] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials/share?null, client=192.168.80.229, traceId=3941237c36724298, method=IterableResponse com.oceanbase.ocp.server.controller.ProfileController.listSharedCredentials(AccessTargetType,Pageable), args=Page request [number: 0, size 2000, sort: createTime: DESC],] 2023-05-15 14:13:31.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a8447a135dc54ed4,37dd2e410430] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST&keyword=, client=192.168.80.229, traceId=a8447a135dc54ed4, method=PaginatedResponse com.oceanbase.ocp.server.controller.ProfileController.listCredentials(AccessTargetType,String,Pageable), args=HOST,,Page request [number: 0, size 2000, sort: create_time: DESC],] 2023-05-15 14:13:31.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,c3aabe9fb1004a04,92d42d898ebe] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=192.168.80.229, traceId=c3aabe9fb1004a04, method=PaginatedResponse com.oceanbase.ocp.server.controller.ComputeController.listHosts(List,String,List,List,List,List,Pageable), args=Page request [number: 0, size 2000, sort: id: DESC],] 2023-05-15 14:13:31.489 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c3702e7be870410c,12024db9d958] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=192.168.80.229, traceId=c3702e7be870410c, method=PaginatedResponse com.oceanbase.ocp.server.controller.ProfileController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 2000, sort: create_time: DESC],] 2023-05-15 14:13:31.492 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,5d66923db4b64660,29f6ea8446d5] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/user/listAll client=192.168.80.229, traceId=5d66923db4b64660, duration=5 ms] 2023-05-15 14:13:31.496 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,3f8024e0c7364fb4,c845a6be3e2a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=192.168.80.229, traceId=3f8024e0c7364fb4, duration=9 ms] 2023-05-15 14:13:31.496 WARN 11 --- [http-nio-0.0.0.0-8080-exec-60,c3aabe9fb1004a04,92d42d898ebe] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2023-05-15 14:13:31.502 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,a8447a135dc54ed4,37dd2e410430] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=192.168.80.229, traceId=a8447a135dc54ed4, duration=14 ms] 2023-05-15 14:13:31.502 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,c3702e7be870410c,12024db9d958] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=192.168.80.229, traceId=c3702e7be870410c, duration=13 ms] 2023-05-15 14:13:31.510 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,43f388c72f554b6e,2fdbd6446f93] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=OB&keyword=, client=192.168.80.229, traceId=43f388c72f554b6e, method=PaginatedResponse com.oceanbase.ocp.server.controller.ProfileController.listCredentials(AccessTargetType,String,Pageable), args=OB,,Page request [number: 0, size 2000, sort: create_time: DESC],] 2023-05-15 14:13:31.515 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,43f388c72f554b6e,2fdbd6446f93] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=192.168.80.229, traceId=43f388c72f554b6e, duration=6 ms] 2023-05-15 14:13:31.523 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,c3aabe9fb1004a04,92d42d898ebe] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=192.168.80.229, traceId=c3aabe9fb1004a04, duration=35 ms] 2023-05-15 14:13:31.526 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,3941237c36724298,160b91699fdb] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials/share client=192.168.80.229, traceId=3941237c36724298, duration=38 ms] 2023-05-15 14:13:31.554 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132006, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:31.557 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132006, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:13:31.551+08:00, endTime=null} 2023-05-15 14:13:31.569 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132007, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:31.571 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132007, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:13:31.567+08:00, endTime=null} 2023-05-15 14:13:31.577 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132006, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116390, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:31.551+08:00, sub_task_instance_id=132006}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:13:31.580 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132005, operation:EXECUTE, state: RUNNING 2023-05-15 14:13:31.582 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132005, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:13:31.578+08:00, endTime=null} 2023-05-15 14:13:31.590 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01e2d47cb87b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132007, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116391, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:31.567+08:00, sub_task_instance_id=132007}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:13:31.590 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01e2d47cb87b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:13:31.601 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132005, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116390, task_operation=execute, latest_execution_start_time=2023-05-15T14:13:31.578+08:00, sub_task_instance_id=132005}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:13:31.609 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:13:31.619 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.620 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.620 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.620 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.621 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.621 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.621 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.621 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.622 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.622 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.622 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.622 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.623 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:13:31.624 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.624 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.624 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.625 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:13:31.625 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.625 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.626 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:13:31.626 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:13:31.626 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:13:31.626 INFO 11 --- [pool-3247397-thread-2,fb27d19334114ed7,72983a824afc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:13:31.626 INFO 11 --- [pool-3247397-thread-3,fb27d19334114ed7,4ec8aec083c6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:13:31.627 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.627 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.627 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.627 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:13:31.628 INFO 11 --- [pool-3247397-thread-4,fb27d19334114ed7,a23ba263c7ec] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:13:31.628 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.628 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.628 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.628 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.629 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.629 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:13:31.629 INFO 11 --- [pool-3247397-thread-1,fb27d19334114ed7,6c2d4c68da95] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:13:31.632 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.633 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.633 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.633 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.633 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:13:31.634 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.637 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.638 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.639 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.639 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.639 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:13:31.640 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.640 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:13:31.640 INFO 11 --- [pool-3247397-thread-6,fb27d19334114ed7,bbc3c0c1c020] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:13:31.640 INFO 11 --- [pool-3247397-thread-5,fb27d19334114ed7,1e17a5c3f289] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:13:31.640 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:13:31.641 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01e2d47cb87b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:13:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.643 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.644 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:13:31.645 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,01e2d47cb87b] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132007, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:31.648 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.648 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.648 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.650 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.650 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.650 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.660 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.660 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.661 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.664 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.664 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.664 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.677 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.680 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.680 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.680 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.688 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.689 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.689 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.694 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.694 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.695 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.699 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.699 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.699 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:13:31.709 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.709 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.709 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:13:31.709 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:13:31.718 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.718 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.718 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:13:31.730 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.730 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.730 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:13:31.734 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.735 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.735 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.735 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.736 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.739 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:13:31.740 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.740 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.741 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.754 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.754 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.754 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.755 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.755 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.755 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.755 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.756 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.760 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.760 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.761 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.764 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.766 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.766 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.767 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.781 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.782 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.782 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.782 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.783 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.784 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.784 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.784 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.786 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.787 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.787 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.788 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.792 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.793 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.793 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.795 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.795 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.796 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.796 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.796 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.797 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.797 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.797 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.807 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.808 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.808 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.808 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.809 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.812 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.813 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.814 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.814 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:13:31.814 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.814 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.816 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:13:31.823 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.823 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.823 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.823 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.824 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.825 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.825 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.825 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.839 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.840 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.840 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.840 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.840 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.840 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.841 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.841 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.841 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.841 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.842 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.842 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.842 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:13:31.847 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.847 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.848 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.852 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.853 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.853 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.853 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.853 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.854 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.855 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.855 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:13:31.855 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:13:31.858 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:13:31.861 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.863 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:13:31.863 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:31.864 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.864 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.875 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:31.876 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.876 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.876 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.876 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.876 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:31.878 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:13:31.878 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:13:31.878 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:13:31.879 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.879 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.879 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:13:31.882 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,63a3bf3a0e7c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132006, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:31.888 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.889 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:13:31.890 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.890 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.890 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:13:31.898 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.901 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:13:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:13:31.910 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.913 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:13:31.913 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:31.913 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:31.913 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:13:31.929 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:13:31.932 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:13:31.932 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:13:31.932 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:13:31.939 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,d22b14b7e346] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132005, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:13:31.984 INFO 11 --- [pool-ocp-schedules-8,,fc20e1eae542] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:31.984 INFO 11 --- [pool-ocp-schedules-8,,fc20e1eae542] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:32.063 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116390, status:SUCCESS 2023-05-15 14:13:32.064 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116390, status:SUCCESS 2023-05-15 14:13:32.075 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116391, status:SUCCESS 2023-05-15 14:13:32.076 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116391, status:SUCCESS 2023-05-15 14:13:32.081 INFO 11 --- [pool-ocp-schedules-20,,971ec016c709] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:13:32.083 INFO 11 --- [pool-ocp-schedules-20,,971ec016c709] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:13:32.089 INFO 11 --- [pool-ocp-schedules-4,,3d29920b54a8] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:13:32.091 INFO 11 --- [pool-ocp-schedules-4,,3d29920b54a8] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:13:32.100 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.101 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.101 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.101 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.102 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.102 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.102 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.102 INFO 11 --- [ForkJoinPool-66084-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:13:32.103 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.104 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.104 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.104 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.106 INFO 11 --- [pool-ocp-schedules-12,,7c906c835205] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:13:32.108 INFO 11 --- [pool-ocp-schedules-12,,7c906c835205] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:13:32.109 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.109 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.109 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.113 INFO 11 --- [ForkJoinPool-66084-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:13:32.130 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.131 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.131 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.131 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.132 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.132 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.133 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:13:32.150 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.151 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.151 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.152 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.156 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.156 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.157 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:13:32.176 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:32.176 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.176 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.176 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.176 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.177 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.177 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.177 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:13:32.178 WARN 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:13:32.179 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.179 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.179 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:13:32.180 INFO 11 --- [ForkJoinPool-66086-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:13:32.180 INFO 11 --- [ForkJoinPool-66086-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:13:32.180 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:13:32.188 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.188 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:13:32.198 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.210 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:32.211 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.211 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.211 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.211 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.212 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:13:32.212 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.212 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.223 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:13:32.223 INFO 11 --- [ForkJoinPool-66086-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:13:32.228 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:32.229 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.229 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.230 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.230 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.235 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.235 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.235 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:13:32.238 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.238 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.239 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:13:32.241 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:13:32.260 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.261 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:13:32.279 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.279 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:13:32.296 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.296 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:13:32.315 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.315 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:13:32.333 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.333 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:13:32.354 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:13:32.373 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:13:32.374 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.375 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.375 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.376 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.381 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:13:32.381 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:13:32.381 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:13:32.400 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,f6ec42fcfaaf4914,c81af51ee147] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=f6ec42fcfaaf4914, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:32.405 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,f6ec42fcfaaf4914,c81af51ee147] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=f6ec42fcfaaf4914, duration=6 ms] 2023-05-15 14:13:32.406 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:13:32.406 INFO 11 --- [ForkJoinPool-66086-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:13:33.005 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:33.006 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:34.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:34.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:34.830199+08:00, lastProcessTime=2023-05-15T14:13:24.831241+08:00 2023-05-15 14:13:34.863 INFO 11 --- [alarm-detect-7,,01f692e40a1b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131194 ~ 1684131204 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.265, values={1684131200=246.265}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:34.872 INFO 11 --- [alarm-detect-7,,620b172c5991] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131144 ~ 1684131204 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389577.938, values={1684131180=389577.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:13:34.873 INFO 11 --- [alarm-detect-7,,620b172c5991] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131144 ~ 1684131204 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994375.107, values={1684131180=994375.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:34.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:34.830199+08:00 2023-05-15 14:13:36.007 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:36.009 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:36.192 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:36.208 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:36.225 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:36.241 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:37.137 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:38.276 INFO 11 --- [pool-overview-container-4,,bda207bfe45e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:38.521 INFO 11 --- [pool-scanner--15977427-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:38.998 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,,] c.o.o.c.s.util.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/batchValidate, client 192.168.80.229, traceId ] 2023-05-15 14:13:39.004 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d33d58f80e4d40,88f47c17f954] c.o.o.server.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/batchValidate?null, client=192.168.80.229, traceId=00d33d58f80e4d40, method=IterableResponse com.oceanbase.ocp.server.controller.ProfileController.batchValidateCredential(BatchValidateCredentialRequest), args=BatchValidateCredentialRequest(credentialIdList=[20]),] 2023-05-15 14:13:39.010 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:39.011 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:39.032 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d33d58f80e4d40,88f47c17f954] c.o.o.s.p.v.CredentialValidateExecutor : Begin to validate credential, name:5f759ddd-5c14-40bc-91a2-646178c5df08, type:OB 2023-05-15 14:13:39.033 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d33d58f80e4d40,88f47c17f954] c.o.o.s.p.v.CredentialValidateExecutor : Await validate all credentials, size:1 2023-05-15 14:13:39.091 INFO 11 --- [validate-credential-thread,00d33d58f80e4d40,89f3d1a14f2f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] invalidate ob connector, clusterName=obcluster, tenantName=test_tenant, username=root 2023-05-15 14:13:39.091 INFO 11 --- [validate-credential-thread,00d33d58f80e4d40,89f3d1a14f2f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=obcluster, obClusterId=null, tenantName=test_tenant, username=root, address=10.xxx.xx.11, port=2981, database=oceanbase) 2023-05-15 14:13:39.091 INFO 11 --- [validate-credential-thread,00d33d58f80e4d40,89f3d1a14f2f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=10.xxx.xx.11, port=2981, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:13:39.098 INFO 11 --- [obsdk-init-thread-5,00d33d58f80e4d40,50beb17e3ea5] com.alibaba.druid.pool.DruidDataSource : {dataSource-104,ob-connector-datasource_direct_test_tenant_root_10.168.89.11_2981} inited 2023-05-15 14:13:39.098 INFO 11 --- [obsdk-init-thread-5,00d33d58f80e4d40,50beb17e3ea5] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.xxx.xx.11, port=2981, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:13:39.098 INFO 11 --- [validate-credential-thread,00d33d58f80e4d40,89f3d1a14f2f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2023-05-15 14:13:39.098 INFO 11 --- [validate-credential-thread,00d33d58f80e4d40,89f3d1a14f2f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.11, sql port:2981 2023-05-15 14:13:39.099 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d33d58f80e4d40,88f47c17f954] c.o.o.s.p.v.CredentialValidateExecutor : Await validate all credential success 2023-05-15 14:13:39.099 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,00d33d58f80e4d40,88f47c17f954] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/batchValidate client=192.168.80.229, traceId=00d33d58f80e4d40, duration=99 ms] 2023-05-15 14:13:39.219 INFO 11 --- [pool-ocp-schedules-6,,e38b3f5bebc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:40.392 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,a309a9eb505b4ad0,98243a676c91] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=a309a9eb505b4ad0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:40.398 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,a309a9eb505b4ad0,98243a676c91] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=a309a9eb505b4ad0, duration=6 ms] 2023-05-15 14:13:42.011 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:42.012 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:42.509 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:44.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:44.832429+08:00, lastProcessTime=2023-05-15T14:13:34.830199+08:00 2023-05-15 14:13:44.858 INFO 11 --- [alarm-detect-5,,b7a167add58c] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131204 ~ 1684131214 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.31, values={1684131210=246.31}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:44.890 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:44.832429+08:00 2023-05-15 14:13:45.012 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:45.013 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:45.525 INFO 11 --- [pool-ocp-schedules-12,,3dc6c2c7c5ff] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514524348172509184 2023-05-15 14:13:45.527 INFO 11 --- [pool-ocp-schedules-12,,3dc6c2c7c5ff] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, newSeriesIdsCount=4, maxSyncedSeriesId=514606205178257408 2023-05-15 14:13:46.195 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:46.211 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:46.228 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:46.246 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:47.141 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:48.013 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:48.014 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:48.274 INFO 11 --- [pool-overview-container-8,,2fb28bb58563] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:48.521 INFO 11 --- [pool-scanner--15977527-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:49.241 INFO 11 --- [pool-ocp-schedules-12,,512b2fa1fa53] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:51.014 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:51.016 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:51.902 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:51.915 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:51.918 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:51.989 INFO 11 --- [pool-ocp-schedules-4,,3b1627b0935e] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:51.990 INFO 11 --- [pool-ocp-schedules-4,,3b1627b0935e] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:13:52.797 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,ade7b6920bf84024,4b57aff23d20] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=ade7b6920bf84024, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:52.802 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,ade7b6920bf84024,4b57aff23d20] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=ade7b6920bf84024, duration=5 ms] 2023-05-15 14:13:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,17388568bd22] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40070} 2023-05-15 14:13:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,17388568bd22] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:13:54.016 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:54.017 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:13:54.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:13:54.831554+08:00, lastProcessTime=2023-05-15T14:13:44.832429+08:00 2023-05-15 14:13:54.858 INFO 11 --- [alarm-detect-12,,58ba123cc22f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131214 ~ 1684131224 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.31, values={1684131220=246.31}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:13:54.892 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:13:54.831554+08:00 2023-05-15 14:13:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,155b43bd0947] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:13:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,a017f5141194] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:13:56.197 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:56.213 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:56.229 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:56.248 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:57.017 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:13:57.018 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:13:57.144 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:57.512 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:13:58.044 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,19f81bc92f5b4e3c,86093e234588] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=19f81bc92f5b4e3c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:13:58.045 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,19f81bc92f5b4e3c,86093e234588] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=19f81bc92f5b4e3c, duration=2 ms] 2023-05-15 14:13:58.274 INFO 11 --- [pool-overview-container-5,,ce781ccbdcdf] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:13:58.522 INFO 11 --- [pool-scanner--15977621-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:13:59.262 INFO 11 --- [pool-ocp-schedules-15,,12f8d3e68004] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:13:59.985 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,87b0a5cfbe2c4001,05d7c4ccd17d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=87b0a5cfbe2c4001, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:13:59.990 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,87b0a5cfbe2c4001,05d7c4ccd17d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=87b0a5cfbe2c4001, duration=6 ms] 2023-05-15 14:14:00.001 INFO 11 --- [pool-ocp-schedules-17,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:14:00.001 INFO 11 --- [pool-ocp-schedules-17,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131240 2023-05-15 14:14:00.001 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:14:00.001 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:14:00.019 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:00.020 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:00.045 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:14:00.586 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:12:01.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.587 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:12:01.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.593 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:14:00.593 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.593 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.593 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:00.602 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:01.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:13:01.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.606 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:14:00.606 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.607 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:00.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:01.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.611 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:13:01.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:14:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:00.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:01.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.625 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:13:01.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:14:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:00.643 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:31.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.643 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:13:31.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.646 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:14:00.646 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.646 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:00.653 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:01.0, nextRun=2023-05-15 14:14:00.0, currentDate=Mon May 15 14:14:00 CST 2023 2023-05-15 14:14:00.653 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:13:01.0, nextRunTime=2023-05-15 14:14:00.0} 2023-05-15 14:14:00.657 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:14:00.658 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:00.658 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:01.001 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132010, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.003 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132010, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:14:00.999+08:00, endTime=null} 2023-05-15 14:14:01.017 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132011, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.020 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132011, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:14:01.014+08:00, endTime=null} 2023-05-15 14:14:01.029 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5d0aef62a22e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132010, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116393, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:00.998+08:00, sub_task_instance_id=132010}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.032 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132008, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.034 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132008, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:14:01.029+08:00, endTime=null} 2023-05-15 14:14:01.037 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:14:01.038 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:14:01.038 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:14:01.038 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:14:01.038 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:14:01.043 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4e7d86bc44d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132011, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116394, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.014+08:00, sub_task_instance_id=132011}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.043 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5d0aef62a22e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132010, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.044 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:14:01.045 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:14:01.045 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:14:01.045 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:14:01.045 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:14:01.048 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132009, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.050 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132009, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:14:01.045+08:00, endTime=null} 2023-05-15 14:14:01.059 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,87e53597ec53] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132008, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116392, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.029+08:00, sub_task_instance_id=132008}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.064 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132014, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.066 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132014, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:14:01.061+08:00, endTime=null} 2023-05-15 14:14:01.072 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4e7d86bc44d0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:14:01.073 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4e7d86bc44d0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:14:01.073 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:14:01.075 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,17217b301380] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132009, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116392, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.045+08:00, sub_task_instance_id=132009}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.075 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:14:01.080 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:14:01.080 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:14:01.080 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:14:01.082 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,87e53597ec53] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132008, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.083 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4e7d86bc44d0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132011, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.083 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132012, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.084 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:14:01.087 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132012, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:14:01.080+08:00, endTime=null} 2023-05-15 14:14:01.092 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:14:01.095 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1d61f282e1c3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132014, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116397, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.061+08:00, sub_task_instance_id=132014}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.097 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132013, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:01.098 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,17217b301380] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132009, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.102 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132013, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:14:01.095+08:00, endTime=null} 2023-05-15 14:14:01.104 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132012, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116395, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.080+08:00, sub_task_instance_id=132012}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.113 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,1d61f282e1c3] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132014, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a8362cbee7aa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132013, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116396, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:01.095+08:00, sub_task_instance_id=132013}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:01.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a8362cbee7aa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:14:01.124 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:14:01.124 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:14:01.125 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:14:01.125 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:14:01.130 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,311335350bd1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132012, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:14:01.144 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:14:01.158 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.158 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.158 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.158 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.158 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.159 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.160 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:14:01.161 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-1,fb27d19334114ed7,ebe583aa2292] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-4,fb27d19334114ed7,48c606b2c898] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:14:01.164 INFO 11 --- [pool-3247566-thread-3,fb27d19334114ed7,dfdd7bb973bc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:14:01.165 INFO 11 --- [pool-3247566-thread-2,fb27d19334114ed7,25fb171b78b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:14:01.166 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.166 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.166 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.167 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.167 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.167 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.168 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.168 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:01.173 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.173 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.174 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:01.174 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:01.174 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:14:01.175 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:14:01.178 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:14:01.178 INFO 11 --- [pool-3247566-thread-5,fb27d19334114ed7,266ddb259346] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:14:01.179 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:14:01.179 INFO 11 --- [pool-3247566-thread-6,fb27d19334114ed7,89816bf0e7c6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:14:01.181 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a8362cbee7aa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:14:01.187 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,a8362cbee7aa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132013, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,8458ed8cf095] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131300, interval=60 2023-05-15 14:14:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,a000efdfb5dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131300, interval=60 2023-05-15 14:14:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,8458ed8cf095] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:14:01.522 INFO 11 --- [pool-prometheus-minute-schedule-2,,8458ed8cf095] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131241, count=8, elapsed=1682447110281 2023-05-15 14:14:01.538 INFO 11 --- [pool-prometheus-minute-schedule-1,,a000efdfb5dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:14:01.538 INFO 11 --- [pool-prometheus-minute-schedule-1,,a000efdfb5dc] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131241, count=6, elapsed=1682447110297 2023-05-15 14:14:01.623 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116392, status:SUCCESS 2023-05-15 14:14:01.624 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116392, status:SUCCESS 2023-05-15 14:14:01.634 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116393, status:SUCCESS 2023-05-15 14:14:01.635 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116393, status:SUCCESS 2023-05-15 14:14:01.645 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116394, status:SUCCESS 2023-05-15 14:14:01.646 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116394, status:SUCCESS 2023-05-15 14:14:01.657 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116395, status:SUCCESS 2023-05-15 14:14:01.659 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116395, status:SUCCESS 2023-05-15 14:14:01.669 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116396, status:SUCCESS 2023-05-15 14:14:01.670 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116396, status:SUCCESS 2023-05-15 14:14:01.681 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116397, status:SUCCESS 2023-05-15 14:14:01.682 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116397, status:SUCCESS 2023-05-15 14:14:03.020 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:03.021 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:04.714 INFO 11 --- [pool-ocp-schedules-12,,cc39f0b60c22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:14:04.722 INFO 11 --- [pool-ocp-schedules-12,,cc39f0b60c22] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:14:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:04.856 INFO 11 --- [alarm-detect-11,,aef3db064cc1] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131224 ~ 1684131234 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.31, values={1684131230=246.31}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:04.859 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2023-05-15 14:14:04.865 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2023-05-15 14:14:04.869 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:04.867864+08:00, lastProcessTime=2023-05-15T14:13:54.831554+08:00 2023-05-15 14:14:04.908 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:04.867864+08:00 2023-05-15 14:14:05.786 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:06.0, nextRun=2023-05-15 14:14:05.0, currentDate=Mon May 15 14:14:05 CST 2023 2023-05-15 14:14:05.786 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:13:06.0, nextRunTime=2023-05-15 14:14:05.0} 2023-05-15 14:14:05.790 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:14:05.790 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:05.790 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:06.021 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:06.022 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:06.173 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132015, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:06.176 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132015, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:14:06.171+08:00, endTime=null} 2023-05-15 14:14:06.193 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,61d92914b663] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132015, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116398, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:06.171+08:00, sub_task_instance_id=132015}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:06.200 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:06.213 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,8bce3a037243] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:14:06.213 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,ebcbe3b741b3] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:14:06.214 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,5a1878e89121] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:14:06.215 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:06.231 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:06.249 INFO 11 --- [pool-Check obproxy8,fb27d19334114ed7,5a1878e89121] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:14:06.250 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:06.253 INFO 11 --- [pool-Check obproxy7,fb27d19334114ed7,8bce3a037243] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:14:06.256 INFO 11 --- [pool-Check obproxy9,fb27d19334114ed7,ebcbe3b741b3] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:14:06.269 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,61d92914b663] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132015, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:06.786 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116398, status:SUCCESS 2023-05-15 14:14:06.787 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116398, status:SUCCESS 2023-05-15 14:14:07.147 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:08.278 INFO 11 --- [pool-overview-container-3,,f423b525b562] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:08.278 INFO 11 --- [pool-overview-container-3,,f423b525b562] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:08.527 INFO 11 --- [pool-scanner--15977714-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:14:09.022 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:09.023 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:09.281 INFO 11 --- [pool-ocp-schedules-14,,1862d59ea1ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:14:10.587 INFO 11 --- [pool-ocp-schedules-5,,7f6ff2199093] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:10.817 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,378104a478284a66,864ff1a55b68] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=378104a478284a66, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:10.824 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,378104a478284a66,864ff1a55b68] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=378104a478284a66, duration=7 ms] 2023-05-15 14:14:11.916 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131180, nodeCount=1000, affectRows=1000 2023-05-15 14:14:11.924 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131180, nodeCount=1000, validCount=1000 2023-05-15 14:14:11.966 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131180, nodeCount=1000, affectRows=1000 2023-05-15 14:14:11.974 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131180, nodeCount=1000, validCount=1000 2023-05-15 14:14:11.994 INFO 11 --- [pool-ocp-schedules-16,,81f75d05387b] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:11.994 INFO 11 --- [pool-ocp-schedules-16,,81f75d05387b] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:12.004 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131180, nodeCount=690, affectRows=690 2023-05-15 14:14:12.007 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131180, nodeCount=690, validCount=690 2023-05-15 14:14:12.024 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:12.025 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:12.514 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:13.532 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,493c9e9fc0304185,5f3673ef9a77] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=493c9e9fc0304185, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:13.538 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,493c9e9fc0304185,5f3673ef9a77] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=493c9e9fc0304185, duration=6 ms] 2023-05-15 14:14:14.290 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,119b9ea9377a4d70,b9c69a8fd5bc] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=119b9ea9377a4d70, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:14.296 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,119b9ea9377a4d70,b9c69a8fd5bc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=119b9ea9377a4d70, duration=7 ms] 2023-05-15 14:14:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:14.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:14.833057+08:00, lastProcessTime=2023-05-15T14:14:04.867864+08:00 2023-05-15 14:14:14.859 INFO 11 --- [alarm-detect-2,,178069b6eb04] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131234 ~ 1684131244 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.31, values={1684131240=246.31}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:14.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:14.833057+08:00 2023-05-15 14:14:15.025 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:15.026 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:16.210 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:16.217 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:16.233 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:16.253 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:17.150 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:18.026 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:18.028 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:18.274 INFO 11 --- [pool-overview-container-7,,b6e67b4c79c0] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:18.522 INFO 11 --- [pool-scanner--15977814-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:19.304 INFO 11 --- [pool-ocp-schedules-18,,417bd42d11c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:14:20.218 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:20.0, nextRun=2023-05-15 14:14:20.0, currentDate=Mon May 15 14:14:20 CST 2023 2023-05-15 14:14:20.218 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:13:20.0, nextRunTime=2023-05-15 14:14:20.0} 2023-05-15 14:14:20.221 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:14:20.221 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:20.221 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:20.387 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132016, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:20.389 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132016, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:14:20.384+08:00, endTime=null} 2023-05-15 14:14:20.410 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,2d2aca424c8a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132016, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116399, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:20.384+08:00, sub_task_instance_id=132016}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:20.413 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,2d2aca424c8a] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:14:20.413 INFO 11 --- [ForkJoinPool-66089-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:14:20.413 INFO 11 --- [ForkJoinPool-66089-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:14:20.418 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,2d2aca424c8a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132016, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:21.028 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:21.029 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:21.085 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116399, status:SUCCESS 2023-05-15 14:14:21.086 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116399, status:SUCCESS 2023-05-15 14:14:21.904 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:21.917 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:21.920 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:23.026 INFO 11 --- [pool-scanner--15977815-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:23.034 INFO 11 --- [pool-ocp-schedules-9,,4d5b7856c681] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:14:23.034 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:14:23.034 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:14:23.035 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:14:23.035 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:14:23.035 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:14:23.040 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:14:23.041 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:14:23.041 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:14:23.042 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:14:23.042 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:14:23.050 INFO 11 --- [pool-scanner--15977816-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:23.067 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:14:23.068 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:14:23.069 INFO 11 --- [pool-scanner--15977817-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:23.070 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:14:23.069637770+08:00 2023-05-15 14:14:23.070 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:14:23.069716560+08:00 2023-05-15 14:14:23.070 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:14:23.070+08:00 2023-05-15 14:14:23.070 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:14:23.070+08:00 2023-05-15 14:14:23.071 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:14:23.071 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:14:23.072 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:14:23.073 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:14:23.072888627+08:00 2023-05-15 14:14:23.073 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:14:23.073+08:00 2023-05-15 14:14:23.073 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:14:23.073302026+08:00 2023-05-15 14:14:23.074 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:14:23.073+08:00 2023-05-15 14:14:23.074 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:14:23.070+08:00 and status AVAILABLE 2023-05-15 14:14:23.075 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:14:23.070+08:00 and status AVAILABLE 2023-05-15 14:14:23.075 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:14:23.074937440+08:00 2023-05-15 14:14:23.075 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:14:23.075+08:00 2023-05-15 14:14:23.077 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:14:23.073+08:00 and status AVAILABLE 2023-05-15 14:14:23.078 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:14:23.073+08:00 and status AVAILABLE 2023-05-15 14:14:23.079 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:14:23.075+08:00 and status AVAILABLE 2023-05-15 14:14:23.080 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:14:23.080 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:14:23.118 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131263 0 0 , error= 2023-05-15 14:14:23.120 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131263 0 0 , error= 2023-05-15 14:14:23.120 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:14:23.121 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:14:23.121+08:00 2023-05-15 14:14:23.121 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:14:23.121+08:00 2023-05-15 14:14:23.142 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131263 0 0 , error= 2023-05-15 14:14:23.142 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:14:23.121+08:00 and status AVAILABLE 2023-05-15 14:14:23.142 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:14:23.142+08:00 2023-05-15 14:14:23.143 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:14:23.121+08:00 and status AVAILABLE 2023-05-15 14:14:23.144 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:14:23.144 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:14:23.145 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.145 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:14:23.146 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:14:23.166 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131263 0 0 , error= 2023-05-15 14:14:23.170 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131263 0 0 , error= 2023-05-15 14:14:23.170 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:14:23.168+08:00 2023-05-15 14:14:23.170 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:14:23.142+08:00 and status AVAILABLE 2023-05-15 14:14:23.170 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:14:23.170+08:00 2023-05-15 14:14:23.173 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:14:23.174 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.175 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:14:23.168+08:00 and status AVAILABLE 2023-05-15 14:14:23.175 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:14:23.170+08:00 and status AVAILABLE 2023-05-15 14:14:23.178 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:14:23.178 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:14:23.179 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:14:23.179 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:14:23.186 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:14:23.189 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:14:23.189 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:14:23.208 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:14:23.208 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:14:23.212 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:14:23.214 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.216 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:14:23.218 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:14:23.218 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.236 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.240 INFO 11 --- [pool-ocp-schedules-17,,8212f27ce24b] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:23.259 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:14:23.264 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:14:23.264 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:14:23.272 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:14:23.273 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:14:23.275 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:14:23.280 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:14:23.285 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:14:23.295 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:14:23.297 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:14:23.301 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:14:23.303 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:14:23.304 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:14:23.315 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:14:23.333 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:14:23.353 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:14:23.355 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:14:23.355 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:14:23.356 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:14:23.369 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:14:23.370 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:14:23.370 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:14:23.371 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:14:23.398 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:14:23.399 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:14:23.399 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:14:23.399 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:14:23.408 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:14:23.422 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:14:23.442 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:14:23.448 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:14:23.448 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:14:23.460 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:14:23.461 INFO 11 --- [pool-Check host status10,,045eaf2cfb96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:14:23.461 INFO 11 --- [pool-Check host status9,,992737fd07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:14:23.462 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:14:23.473 INFO 11 --- [pool-Check host status22,,f30455493713] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:14:23.475 INFO 11 --- [pool-Check host status26,,1e36c08111fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:14:23.530 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:14:23.545 INFO 11 --- [pool-Check host status12,,6bc626277fcf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:14:23.566 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:14:23.567 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:14:23.567 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:14:23.567 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:14:23.567 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:14:23.573 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:14:23.574 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:14:23.574 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:14:23.574 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:14:23.575 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:14:23.603 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:14:23.605 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:14:23.606 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:14:23.606 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:14:23.606 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:14:23.623 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:14:23.623 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:14:23.623 INFO 11 --- [pool-Check host status18,,fc096d0de27e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:14:23.623 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:14:23.623 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:14:23.624 INFO 11 --- [pool-Check host status25,,f098610c6239] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status15,,abb2e6b169a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:14:23.627 INFO 11 --- [pool-Check host status24,,393347a08ec6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:14:23.628 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:14:23.628 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:14:23.628 INFO 11 --- [pool-Check host status16,,48e32c17f3d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:14:24.029 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:24.030 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:24.835 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:24.835080+08:00, lastProcessTime=2023-05-15T14:14:14.833057+08:00 2023-05-15 14:14:24.925 INFO 11 --- [alarm-detect-5,,2a6742465f07] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131244 ~ 1684131254 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.31, values={1684131250=246.31}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:24.928 INFO 11 --- [alarm-detect-15,,d8e6ff21e039] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131194 ~ 1684131254 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389637.988, values={1684131240=389637.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:14:24.929 INFO 11 --- [alarm-detect-15,,d8e6ff21e039] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131194 ~ 1684131254 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994435.358, values={1684131240=994435.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:24.931 INFO 11 --- [alarm-detect-15,,d8e6ff21e039] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:14:24.965 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:24.835080+08:00 2023-05-15 14:14:25.372 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:26.0, nextRun=2023-05-15 14:14:25.0, currentDate=Mon May 15 14:14:25 CST 2023 2023-05-15 14:14:25.372 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:13:26.0, nextRunTime=2023-05-15 14:14:25.0} 2023-05-15 14:14:25.383 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:14:25.383 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:25.383 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:25.471 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132017, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:25.473 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132017, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:14:25.467+08:00, endTime=null} 2023-05-15 14:14:25.498 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132017, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116400, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:25.467+08:00, sub_task_instance_id=132017}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:14:25.536 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:14:25.537 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:14:25.537 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:14:25.538 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:14:25.538 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:14:25.550 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,c2cb2ae7f7bd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132017, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:26.213 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:26.219 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:26.235 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:26.239 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116400, status:SUCCESS 2023-05-15 14:14:26.241 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116400, status:SUCCESS 2023-05-15 14:14:26.255 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:27.030 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:27.032 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:27.153 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:27.516 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:28.275 INFO 11 --- [pool-overview-container-1,,7486f9c52eb2] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:28.276 INFO 11 --- [pool-overview-container-1,,7486f9c52eb2] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:28.521 INFO 11 --- [pool-scanner--15977915-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:29.131 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b42fe0d1c0664ddc,ffa893109d99] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=b42fe0d1c0664ddc, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:29.137 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b42fe0d1c0664ddc,ffa893109d99] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=b42fe0d1c0664ddc, duration=6 ms] 2023-05-15 14:14:29.324 INFO 11 --- [pool-ocp-schedules-14,,bf423656f96a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:14:30.033 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:30.038 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:30.577 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:13:31.0, nextRun=2023-05-15 14:14:30.0, currentDate=Mon May 15 14:14:30 CST 2023 2023-05-15 14:14:30.577 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:13:31.0, nextRunTime=2023-05-15 14:14:30.0} 2023-05-15 14:14:30.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:14:30.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:30.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:30.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:30.595 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:14:30.0, currentDate=Mon May 15 14:14:30 CST 2023 2023-05-15 14:14:30.595 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:14:30.0} 2023-05-15 14:14:30.598 INFO 11 --- [pool-ocp-schedules-4,,9e4e1cff6bc0] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:30.599 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:14:30.599 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:14:30.599 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:14:31.618 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132018, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:31.627 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132018, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:14:31.607+08:00, endTime=null} 2023-05-15 14:14:31.642 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132019, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:31.649 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132019, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:14:31.639+08:00, endTime=null} 2023-05-15 14:14:31.660 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132020, operation:EXECUTE, state: RUNNING 2023-05-15 14:14:31.662 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132020, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:14:31.657+08:00, endTime=null} 2023-05-15 14:14:31.663 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132018, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116401, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:31.603+08:00, sub_task_instance_id=132018}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:14:31.691 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c922b8d69cde] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132020, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116402, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:31.657+08:00, sub_task_instance_id=132020}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:14:31.691 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c922b8d69cde] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:14:31.692 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132019, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116401, task_operation=execute, latest_execution_start_time=2023-05-15T14:14:31.639+08:00, sub_task_instance_id=132019}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:14:31.710 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:14:31.712 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:14:31.713 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:14:31.713 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:14:31.716 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:14:31.719 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.721 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:14:31.721 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.723 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.723 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.724 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:14:31.725 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.726 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.726 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.726 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.733 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.734 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.734 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.734 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.737 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.746 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.747 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.747 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:31.751 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.751 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.755 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:14:31.755 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.760 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.760 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.762 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:14:31.762 INFO 11 --- [pool-3247724-thread-1,fb27d19334114ed7,458a56b5b328] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:14:31.764 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.764 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.764 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.764 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.764 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.765 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:31.765 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.765 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.765 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:14:31.769 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.771 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:14:31.772 INFO 11 --- [pool-3247724-thread-3,fb27d19334114ed7,82a54ced74e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:14:31.773 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.773 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.779 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:14:31.779 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.779 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.780 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.780 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:31.780 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.780 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.780 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.780 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:14:31.781 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.783 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:14:31.783 INFO 11 --- [pool-3247724-thread-4,fb27d19334114ed7,9465fbafecb9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:14:31.786 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.786 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.786 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.787 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.787 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.788 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:14:31.788 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.788 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:14:31.789 INFO 11 --- [pool-3247724-thread-5,fb27d19334114ed7,23e5defafb93] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:14:31.791 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.791 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.791 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:31.793 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.793 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.795 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:14:31.799 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:14:31.799 INFO 11 --- [pool-3247724-thread-2,fb27d19334114ed7,fb5c4ea8f8b1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:14:31.800 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.800 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.800 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:31.807 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.807 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.809 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:14:31.832 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.833 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.833 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:14:31.842 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.842 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.842 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.843 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.847 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.847 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.848 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:14:31.857 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.857 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.857 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:14:31.866 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:14:31.872 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.873 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.873 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:14:31.879 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:14:31.879 INFO 11 --- [pool-3247724-thread-6,fb27d19334114ed7,528728fae865] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:14:31.880 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c922b8d69cde] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:14:31.884 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.884 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.885 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:31.886 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c922b8d69cde] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132020, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:31.887 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.888 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.888 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.888 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.888 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.889 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.889 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.890 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.902 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.903 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:31.903 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.904 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.904 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.904 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:31.916 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.916 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.917 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:31.925 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.926 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.926 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.926 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.926 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.927 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.927 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.928 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:31.929 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:14:31.942 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.943 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.943 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.943 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.943 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.945 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.945 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.945 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:31.966 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.967 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.967 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.967 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.968 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.970 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.970 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.970 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.970 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.971 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.972 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.973 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.973 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:31.974 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.974 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.975 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:31.988 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.989 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.989 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.989 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.990 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.991 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:31.991 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.991 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:31.995 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:31.996 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.997 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:31.997 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:31.998 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:31.999 INFO 11 --- [pool-ocp-schedules-5,,dad80d97be8f] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:31.999 INFO 11 --- [pool-ocp-schedules-5,,dad80d97be8f] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:32.002 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.002 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.003 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:32.014 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.015 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.015 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.015 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.016 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.017 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.017 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.018 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:32.032 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.033 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.033 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.033 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.034 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.035 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.035 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.036 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:32.036 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:14:32.036 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.040 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:14:32.042 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.042 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.043 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.065 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.065 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.066 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.066 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.066 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.070 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.070 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.071 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:32.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.075 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:14:32.076 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,0cdbcbb6a59a4eea,42414d1716b2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=0cdbcbb6a59a4eea, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:14:32.078 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,0cdbcbb6a59a4eea,42414d1716b2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=0cdbcbb6a59a4eea, duration=2 ms] 2023-05-15 14:14:32.088 INFO 11 --- [pool-ocp-schedules-11,,a6c99903c55b] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:14:32.089 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.091 INFO 11 --- [pool-ocp-schedules-11,,a6c99903c55b] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:14:32.092 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:14:32.092 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.092 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.092 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:14:32.095 INFO 11 --- [pool-ocp-schedules-11,,61a74f43be22] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:14:32.097 INFO 11 --- [pool-ocp-schedules-11,,61a74f43be22] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:14:32.101 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.101 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.102 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.103 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.103 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.104 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:14:32.104 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.104 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.104 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.104 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:14:32.108 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.108 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.109 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:14:32.110 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:14:32.111 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.112 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.112 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.112 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.113 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.113 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:14:32.113 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.113 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.114 INFO 11 --- [ForkJoinPool-66090-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:14:32.114 INFO 11 --- [pool-ocp-schedules-14,,b2eb2ec5057e] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:14:32.116 INFO 11 --- [pool-ocp-schedules-14,,b2eb2ec5057e] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:14:32.117 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.118 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.118 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.118 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.119 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:14:32.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.120 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:32.123 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.123 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.124 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.127 INFO 11 --- [ForkJoinPool-66090-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:14:32.130 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.132 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:14:32.132 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:14:32.132 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:14:32.137 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,5900ae4916fb] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132018, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:32.140 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.140 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.140 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.140 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.141 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.141 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.141 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.141 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:14:32.145 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.146 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.146 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.146 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.154 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.155 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.155 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.155 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.173 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.173 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.174 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:14:32.174 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.174 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.175 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:14:32.189 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.190 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.190 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.190 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.190 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.191 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.192 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:14:32.192 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.192 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.192 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:14:32.193 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.193 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.194 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:14:32.194 WARN 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:14:32.194 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.194 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.195 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:14:32.196 INFO 11 --- [ForkJoinPool-66092-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:14:32.196 INFO 11 --- [ForkJoinPool-66092-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:14:32.196 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:14:32.204 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:14:32.206 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:14:32.206 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:14:32.206 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:14:32.206 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.206 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:14:32.212 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,7481e818c379] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132019, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:14:32.219 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.234 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.234 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.234 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.234 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.235 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.236 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:14:32.236 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.236 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.247 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:14:32.247 INFO 11 --- [ForkJoinPool-66092-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:14:32.250 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.251 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.252 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.252 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.253 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.274 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.274 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.275 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:14:32.279 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.280 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.280 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:14:32.283 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:14:32.302 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.302 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:14:32.316 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.316 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:14:32.333 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.333 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:14:32.351 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.352 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:14:32.369 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.369 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:14:32.369 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116401, status:SUCCESS 2023-05-15 14:14:32.371 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116401, status:SUCCESS 2023-05-15 14:14:32.384 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116402, status:SUCCESS 2023-05-15 14:14:32.385 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116402, status:SUCCESS 2023-05-15 14:14:32.389 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:14:32.407 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:14:32.408 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.408 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.408 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.409 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.414 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:14:32.414 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:14:32.418 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:14:32.438 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:14:32.438 INFO 11 --- [ForkJoinPool-66092-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:14:33.038 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:33.039 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:33.881 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,4b279712fe0e4355,9e0a88466c8a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=4b279712fe0e4355, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:33.886 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,4b279712fe0e4355,9e0a88466c8a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=4b279712fe0e4355, duration=5 ms] 2023-05-15 14:14:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:34.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:34.830283+08:00, lastProcessTime=2023-05-15T14:14:24.835080+08:00 2023-05-15 14:14:34.860 INFO 11 --- [alarm-detect-1,,b4a52bbe761f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131254 ~ 1684131264 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131260=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:34.864 INFO 11 --- [alarm-detect-7,,2c33208a71a6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131204 ~ 1684131264 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389637.938, values={1684131240=389637.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:14:34.865 INFO 11 --- [alarm-detect-7,,2c33208a71a6] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131204 ~ 1684131264 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994435.107, values={1684131240=994435.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:34.867 INFO 11 --- [alarm-detect-7,,2c33208a71a6] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:14:34.891 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:34.830283+08:00 2023-05-15 14:14:36.039 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:36.040 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:36.216 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:36.221 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:36.237 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:36.257 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:37.155 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:38.276 INFO 11 --- [pool-overview-container-6,,3fd0147f3439] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:38.276 INFO 11 --- [pool-overview-container-6,,3fd0147f3439] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:38.521 INFO 11 --- [pool-scanner--15978026-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:39.040 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:39.042 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:39.351 INFO 11 --- [pool-ocp-schedules-1,,a6a73be1f0ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:14:40.168 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,458c8874f1f24d93,78a7d8153eb8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=458c8874f1f24d93, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:40.174 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,458c8874f1f24d93,78a7d8153eb8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=458c8874f1f24d93, duration=6 ms] 2023-05-15 14:14:40.603 INFO 11 --- [pool-ocp-schedules-1,,7c7841d80469] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:14:42.042 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:42.043 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:42.519 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:44.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:44.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:44.828410+08:00, lastProcessTime=2023-05-15T14:14:34.830283+08:00 2023-05-15 14:14:44.858 INFO 11 --- [alarm-detect-9,,7d08835a6781] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131264 ~ 1684131274 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131270=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:44.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:44.828410+08:00 2023-05-15 14:14:45.043 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:45.044 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:45.528 INFO 11 --- [pool-ocp-schedules-12,,ba2730665b30] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514606205178257408 2023-05-15 14:14:46.218 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:46.223 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:46.239 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:46.260 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:47.159 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:47.172 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169} 2023-05-15 14:14:47.196 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.196 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.202 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.202 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.203 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.203 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.204 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.204 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.208 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.208 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.209 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:10:47.118, end=2023-05-15T14:12:47.169},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:14:47.210 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:14:47.210 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:14:48.045 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:48.046 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:48.187 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ce068f9f3ec0467e,b955c2e922cc] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=ce068f9f3ec0467e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:48.192 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,ce068f9f3ec0467e,b955c2e922cc] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=ce068f9f3ec0467e, duration=5 ms] 2023-05-15 14:14:48.275 INFO 11 --- [pool-overview-container-3,,a06449c44cee] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:48.520 INFO 11 --- [pool-scanner--15978126-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:49.374 INFO 11 --- [pool-ocp-schedules-2,,9440496b290a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:14:51.046 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:51.047 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:51.907 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:51.920 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:51.922 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:52.005 INFO 11 --- [pool-ocp-schedules-2,,3be33f1209e6] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:52.005 INFO 11 --- [pool-ocp-schedules-2,,3be33f1209e6] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:14:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,613828f10d40] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40070} 2023-05-15 14:14:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,613828f10d40] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:14:54.023 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.048 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:54.048 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:54.052 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.055 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.058 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.060 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.061 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.075 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.083 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 14 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.106 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 28 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.145 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 125 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.181 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.183 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.185 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.187 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.189 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.190 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.200 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.206 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.214 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.221 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.221 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.223 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.225 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.226 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.228 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.228 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.230 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.231 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.232 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.233 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:08:53.821, end=2023-05-15T14:13:54.02} 2023-05-15 14:14:54.234 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION is disabled or lacking of config 2023-05-15 14:14:54.234 INFO 11 --- [pool-11-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:14:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:14:54.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:14:54.831976+08:00, lastProcessTime=2023-05-15T14:14:44.828410+08:00 2023-05-15 14:14:54.853 INFO 11 --- [alarm-detect-9,,09c9d31c7183] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131274 ~ 1684131284 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131280=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:14:54.888 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:14:54.831976+08:00 2023-05-15 14:14:55.102 INFO 11 --- [pool-prometheus-second-schedule-2,,36fb3c096bb8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:14:55.102 INFO 11 --- [pool-prometheus-second-schedule-2,,a0c00f00c7b6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:14:56.221 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:56.225 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:56.241 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:56.263 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:57.049 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:14:57.049 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:14:57.162 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:57.521 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:14:58.190 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,2a1dbb2b773548fb,ec62bf7a98b7] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=2a1dbb2b773548fb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:14:58.194 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,2a1dbb2b773548fb,ec62bf7a98b7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=2a1dbb2b773548fb, duration=5 ms] 2023-05-15 14:14:58.275 INFO 11 --- [pool-overview-container-4,,f174c790d8cd] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:14:58.521 INFO 11 --- [pool-scanner--15978220-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:14:59.396 INFO 11 --- [pool-ocp-schedules-6,,b1a4704b6659] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-12,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-20,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131300 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-12,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:15:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:15:00.014 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:15:00.014 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:15:00.014 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:15:00.014 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:15:00.020 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:15:00.020 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:15:00.020 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:15:00.020 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:15:00.041 INFO 11 --- [pool-ocp-schedules-12,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:15:00.050 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:00.051 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:00.063 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:15:00.071 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:15:00.072 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:15:00.072 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:15:00.072 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:15:00.072 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:15:00.079 WARN 11 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:15:00.079 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:15:00.563 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.563 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=20, name=Check all cluster configUrl, templateName=CheckClusterConfigUrlSchedule, scheduleRule=0 */5 * * * ?, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.567 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=CheckClusterConfigUrlSchedule, nodeCount=1, taskCount=1 2023-05-15 14:15:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=CheckClusterConfigUrlSchedule, taskDefinitionId=20, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.574 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.575 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=21, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.586 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:15:00.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:15:00.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.604 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=SqlParseTaskSchedule, taskDefinitionId=21, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.609 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.609 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.614 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:15:00.614 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.614 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.619 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.620 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.631 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:15:00.631 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.631 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.636 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.636 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.650 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:15:00.650 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.650 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.655 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:31.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.655 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:14:31.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.659 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:15:00.659 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.659 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.665 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.665 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=35, name=Collect all tenants mini compactions, templateName=CollectTenantMiniCompactionSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.668 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants mini compactions, nodeCount=1, taskCount=1 2023-05-15 14:15:00.668 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.668 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants mini compactions, taskDefinitionId=35, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.675 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:10:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.675 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=37, name=HighRiskSqlSchedule, templateName=HighRiskSqlSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2023-05-15 14:10:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.710 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=HighRiskSqlAuditStatTaskSchedule, nodeCount=1, taskCount=1 2023-05-15 14:15:00.711 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.711 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=HighRiskSqlAuditStatTaskSchedule, taskDefinitionId=37, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:00.718 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:15:00.0, currentDate=Mon May 15 14:15:00 CST 2023 2023-05-15 14:15:00.718 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:15:00.0} 2023-05-15 14:15:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:15:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:00.723 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:01.058 WARN 11 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:15:01.059 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.038934234S 2023-05-15 14:15:01.059 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:15:01.115 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132022, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.117 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132022, name=Sql Parse Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=2023-05-15T14:15:01.106+08:00, endTime=null} 2023-05-15 14:15:01.126 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132023, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.128 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132023, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:15:01.124+08:00, endTime=null} 2023-05-15 14:15:01.134 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,00249bde042d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132022, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116404, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.106+08:00, sub_task_instance_id=132022}, listMap={cluster_ids=[[1,1,1,2]], tenant_ids=[[4,2,3,6]]}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.134 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,00249bde042d] c.o.o.p.sql.internal.task.SqlParseTask : Parse sql for [4, 2, 3, 6] 2023-05-15 14:15:01.141 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132021, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.144 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132021, name=Check root service , state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CheckClusterRootServersTask, seriesId=0, startTime=2023-05-15T14:15:01.139+08:00, endTime=null} 2023-05-15 14:15:01.149 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,49ca24bd39fd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132023, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116405, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.124+08:00, sub_task_instance_id=132023}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.155 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:15:01.155 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:15:01.155 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:15:01.155 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:15:01.156 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:15:01.156 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132026, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.158 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132026, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:15:01.153+08:00, endTime=null} 2023-05-15 14:15:01.160 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:15:01.160 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:15:01.160 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:15:01.161 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:15:01.161 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:15:01.162 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,49ca24bd39fd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132023, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.164 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,50ca9b1d2c23] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132021, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116403, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.138+08:00, sub_task_instance_id=132021}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.166 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,50ca9b1d2c23] c.o.o.s.t.b.c.AbstractAllClusterTask : Check root service executed completed, tryCount=2, successCount=0 2023-05-15 14:15:01.169 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132027, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.171 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,50ca9b1d2c23] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132021, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.172 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132027, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask, seriesId=0, startTime=2023-05-15T14:15:01.167+08:00, endTime=null} 2023-05-15 14:15:01.177 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,279555c48698] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132026, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116408, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.153+08:00, sub_task_instance_id=132026}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.177 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,279555c48698] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:15:01.182 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.182 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.183 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.183 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.184 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132024, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.184 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:15:01.184 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:15:01.185 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132024, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:15:01.181+08:00, endTime=null} 2023-05-15 14:15:01.186 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:15:01.186 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.186 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.186 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.187 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.188 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:15:01.188 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:15:01.189 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:15:01.189 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,00249bde042d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132022, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.190 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:15:01.190 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a2d889a43ec] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132027, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116409, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.167+08:00, sub_task_instance_id=132027}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.190 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a2d889a43ec] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2023-05-15 14:15:01.196 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.196 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.196 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:15:01.197 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132025, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.199 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:15:01.199 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:15:01.199 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:15:01.199 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:15:01.199 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:15:01.200 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:15:01.200 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132025, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:15:01.192+08:00, endTime=null} 2023-05-15 14:15:01.203 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:15:01.204 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.204 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.205 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2023-05-15 14:15:01.206 INFO 11 --- [pool-ocp-schedules-1,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:15:01.206 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:15:01.208 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:15:01.208 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,e8da410a3a51] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132024, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116406, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.181+08:00, sub_task_instance_id=132024}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.212 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.213 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.213 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.213 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.213 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.213 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132028, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.214 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.214 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:15:01.215 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132028, name=High Risk Sql Audit Stat Task, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.HighRiskSqlAuditStatTask, seriesId=0, startTime=2023-05-15T14:15:01.211+08:00, endTime=null} 2023-05-15 14:15:01.216 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:15:01.216 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.216 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2023-05-15 14:15:01.216 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.216 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.217 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:15:01.217 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.219 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.220 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.220 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.220 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.220 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.220 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.221 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.221 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.221 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.221 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.222 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,5249572d48c1414c,c89e0932fda2] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=5249572d48c1414c, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=ocpcluster,1,false,2,] 2023-05-15 14:15:01.223 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:15:01.223 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,5249572d48c1414c,c89e0932fda2] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=5249572d48c1414c, duration=2 ms] 2023-05-15 14:15:01.226 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.226 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.226 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132025, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116407, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.192+08:00, sub_task_instance_id=132025}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.226 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:15:01.226 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:15:01.226 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.226 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.227 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:15:01.228 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.228 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.228 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:15:01.229 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132029, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:01.229 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:15:01.229 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:15:01.230 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.230 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.230 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:15:01.230 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:15:01.230 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132029, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:15:01.226+08:00, endTime=null} 2023-05-15 14:15:01.231 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.231 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.231 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.231 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.233 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:15:01.233 INFO 11 --- [pool-3247893-thread-3,fb27d19334114ed7,340f2f86847b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:15:01.233 INFO 11 --- [pool-3247893-thread-1,fb27d19334114ed7,06698c7af666] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:15:01.233 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,e8da410a3a51] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:15:01.234 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,e8da410a3a51] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:15:01.235 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132028, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116410, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.211+08:00, sub_task_instance_id=132028}, listMap={cluster_ids=[[1,2,1,1]], tenant_ids=[[4,6,2,3]]}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.236 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.t.HighRiskSqlAuditStatTask : start to diagnose[4, 6, 2, 3] 2023-05-15 14:15:01.259 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1684131301236000 2023-05-15 14:15:01.262 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.262 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1684131001876000 2023-05-15 14:15:01.262 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.265 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.265 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:15:01.266 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.266 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.266 INFO 11 --- [pool-subtask-executor29,fb27d19334114ed7,e8da410a3a51] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132024, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.266 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.266 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.267 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:15:01.267 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:01.268 INFO 11 --- [pool-3247893-thread-4,fb27d19334114ed7,7c984705e0b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:15:01.269 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.269 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.269 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,f7d8b7e9ff984eaa,ea1a22d30e66] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&ObClusterId=1&version=2, client=10.xxx.xx.10, traceId=f7d8b7e9ff984eaa, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obcluster,1,false,2,] 2023-05-15 14:15:01.269 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684130701000000, 1684131301000000] 2023-05-15 14:15:01.270 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,f7d8b7e9ff984eaa,ea1a22d30e66] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=f7d8b7e9ff984eaa, duration=1 ms] 2023-05-15 14:15:01.271 INFO 11 --- [pool-3247893-thread-2,fb27d19334114ed7,d1fc03c1e902] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:15:01.271 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:15:01.271 INFO 11 --- [pool-3247893-thread-5,fb27d19334114ed7,aef345ce9f49] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:15:01.272 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:15:01.272 INFO 11 --- [pool-3247893-thread-6,fb27d19334114ed7,b78b235c34a2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:15:01.272 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.272 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.273 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2023-05-15 14:15:01.273 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,279555c48698] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:15:01.274 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:01.274 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.275 INFO 11 --- [ForkJoinPool-66095-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.10, 2982] 2023-05-15 14:15:01.277 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:15:01.277 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:15:01.278 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2023-05-15 14:15:01.278 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:15:01.278 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:15:01.279 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.279 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.280 INFO 11 --- [pool-subtask-executor32,fb27d19334114ed7,279555c48698] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132026, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.280 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1684130701000000, 1684131301000000] 2023-05-15 14:15:01.281 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,651fcccfc2e9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132029, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116411, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:01.226+08:00, sub_task_instance_id=132029}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:01.284 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,12fe091e4953] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132025, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.284 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1684131301280000 2023-05-15 14:15:01.284 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1684131001894000 2023-05-15 14:15:01.296 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,651fcccfc2e9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132029, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.301 INFO 11 --- [pool-ocp-schedules-1,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:15:01.302 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2023-05-15 14:15:01.306 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2currentTimeUs:1684131301303000 2023-05-15 14:15:01.306 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_2diagnosisStartTimeUs:1684131001862000 2023-05-15 14:15:01.335 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_3totalLength:0 2023-05-15 14:15:01.335 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:15:01.346 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:15:01.350 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:15:01.350 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:15:01.350 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:15:01.351 INFO 11 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:15:01.356 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.356 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.357 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,598565a93dc1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132028, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.357 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.11, 2982] 2023-05-15 14:15:01.362 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:15:01.368 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.368 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.371 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.13, 2982] 2023-05-15 14:15:01.374 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:01.375 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:01.376 INFO 11 --- [ForkJoinPool-66095-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.xxx.xx.12, 2982] 2023-05-15 14:15:01.510 INFO 11 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : Started application in 1.495 seconds (JVM running for 600957.484) 2023-05-15 14:15:01.510 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.495986389S 2023-05-15 14:15:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,2b50214dee1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131360, interval=60 2023-05-15 14:15:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,29d92232c742] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131360, interval=60 2023-05-15 14:15:01.534 INFO 11 --- [pool-prometheus-minute-schedule-2,,2b50214dee1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:15:01.534 INFO 11 --- [pool-prometheus-minute-schedule-2,,2b50214dee1d] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131301, count=8, elapsed=1682447170233 2023-05-15 14:15:01.546 INFO 11 --- [pool-prometheus-minute-schedule-1,,29d92232c742] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:15:01.547 INFO 11 --- [pool-prometheus-minute-schedule-1,,29d92232c742] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131301, count=6, elapsed=1682447170246 2023-05-15 14:15:01.558 INFO 11 --- [pool-3247894-thread-1,fb27d19334114ed7,b901418e1a79] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:15:01.593 INFO 11 --- [pool-3247894-thread-2,fb27d19334114ed7,ddf70a9dfd26] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2023-05-15 14:15:01.594 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a2d889a43ec] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2023-05-15 14:15:01.613 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a2d889a43ec] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132027, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:01.736 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:15:01.736 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:15:01.736 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:15:01.736 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:15:02.021 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116403, status:SUCCESS 2023-05-15 14:15:02.022 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116403, status:SUCCESS 2023-05-15 14:15:02.032 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116404, status:SUCCESS 2023-05-15 14:15:02.033 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116404, status:SUCCESS 2023-05-15 14:15:02.041 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116405, status:SUCCESS 2023-05-15 14:15:02.041 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116405, status:SUCCESS 2023-05-15 14:15:02.050 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116406, status:SUCCESS 2023-05-15 14:15:02.051 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116406, status:SUCCESS 2023-05-15 14:15:02.059 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116407, status:SUCCESS 2023-05-15 14:15:02.061 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116407, status:SUCCESS 2023-05-15 14:15:02.072 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116408, status:SUCCESS 2023-05-15 14:15:02.074 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116408, status:SUCCESS 2023-05-15 14:15:02.085 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116409, status:SUCCESS 2023-05-15 14:15:02.086 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116409, status:SUCCESS 2023-05-15 14:15:02.095 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116410, status:SUCCESS 2023-05-15 14:15:02.096 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116410, status:SUCCESS 2023-05-15 14:15:02.113 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116411, status:SUCCESS 2023-05-15 14:15:02.115 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116411, status:SUCCESS 2023-05-15 14:15:02.115 INFO 11 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:15:03.051 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:03.052 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:04.714 INFO 11 --- [pool-ocp-schedules-7,,8e6286c0a11e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:15:04.723 INFO 11 --- [pool-ocp-schedules-7,,8e6286c0a11e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:15:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:04.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:04.832187+08:00, lastProcessTime=2023-05-15T14:14:54.831976+08:00 2023-05-15 14:15:04.859 INFO 11 --- [alarm-detect-8,,1ff78b2c5f6c] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131284 ~ 1684131294 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131290=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:04.894 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:04.832187+08:00 2023-05-15 14:15:05.876 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:06.0, nextRun=2023-05-15 14:15:05.0, currentDate=Mon May 15 14:15:05 CST 2023 2023-05-15 14:15:05.877 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:14:06.0, nextRunTime=2023-05-15 14:15:05.0} 2023-05-15 14:15:05.880 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:15:05.881 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:05.881 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:06.052 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:06.054 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:06.110 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,a356a02cdce3492e,9c55afc3b86c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=a356a02cdce3492e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:15:06.111 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,a356a02cdce3492e,9c55afc3b86c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=a356a02cdce3492e, duration=2 ms] 2023-05-15 14:15:06.224 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:06.228 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:06.243 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:06.265 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:06.305 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132030, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:06.307 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132030, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:15:06.302+08:00, endTime=null} 2023-05-15 14:15:06.326 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,9a54ff8a6661] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132030, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116412, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:06.302+08:00, sub_task_instance_id=132030}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:06.350 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,c4190f3cbce8] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:15:06.351 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,03bb4bc7b0d7] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:15:06.353 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,ca95d927a64f] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:15:06.400 INFO 11 --- [pool-Check obproxy10,fb27d19334114ed7,c4190f3cbce8] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:15:06.406 INFO 11 --- [pool-Check obproxy11,fb27d19334114ed7,ca95d927a64f] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:15:06.413 INFO 11 --- [pool-Check obproxy12,fb27d19334114ed7,03bb4bc7b0d7] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:15:06.428 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,9a54ff8a6661] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132030, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:07.165 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:07.231 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116412, status:SUCCESS 2023-05-15 14:15:07.232 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116412, status:SUCCESS 2023-05-15 14:15:07.779 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f99385e375df4bf0,e49502891600] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=f99385e375df4bf0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:07.785 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,f99385e375df4bf0,e49502891600] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=f99385e375df4bf0, duration=6 ms] 2023-05-15 14:15:08.277 INFO 11 --- [pool-overview-container-8,,6119245d1c11] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:15:08.278 INFO 11 --- [pool-overview-container-8,,6119245d1c11] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:08.530 INFO 11 --- [pool-scanner--15978313-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:15:09.054 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:09.055 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:09.418 INFO 11 --- [pool-ocp-schedules-13,,72300221b382] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:09.951 INFO 11 --- [pool-ocp-schedules-4,,7454a84a1b25] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40070} 2023-05-15 14:15:09.951 INFO 11 --- [pool-ocp-schedules-4,,7454a84a1b25] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:15:10.254 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,7e0a1b6ccde44e53,118fa2d5a888] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=7e0a1b6ccde44e53, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:10.260 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,7e0a1b6ccde44e53,118fa2d5a888] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=7e0a1b6ccde44e53, duration=6 ms] 2023-05-15 14:15:10.619 INFO 11 --- [pool-ocp-schedules-3,,b37f532a260b] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:15:12.011 INFO 11 --- [pool-ocp-schedules-8,,9377087728ac] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:12.011 INFO 11 --- [pool-ocp-schedules-8,,9377087728ac] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:12.055 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:12.057 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:12.069 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131240, nodeCount=1000, affectRows=1000 2023-05-15 14:15:12.077 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131240, nodeCount=1000, validCount=1000 2023-05-15 14:15:12.125 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131240, nodeCount=1000, affectRows=1000 2023-05-15 14:15:12.131 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131240, nodeCount=1000, validCount=1000 2023-05-15 14:15:12.160 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131240, nodeCount=690, affectRows=690 2023-05-15 14:15:12.163 INFO 11 --- [pool-metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131240, nodeCount=690, validCount=690 2023-05-15 14:15:12.524 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:14.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:14.830451+08:00, lastProcessTime=2023-05-15T14:15:04.832187+08:00 2023-05-15 14:15:14.864 INFO 11 --- [alarm-detect-11,,138050cd157a] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131294 ~ 1684131304 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131300=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:14.899 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:14.830451+08:00 2023-05-15 14:15:15.057 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:15.058 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:16.228 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:16.231 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:16.246 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:16.267 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:17.169 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:17.615 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:15:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:17.671 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:17.671 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:17.671 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:15:17.695 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:17.697 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:17.697 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:17.698 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:17.698 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:17.703 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:17.703 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:17.704 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:15:17.713 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:15:17.713 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:15:17.713 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:15:17.713 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:15:17.717 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:13:02, target=2023-05-15T14:10:47, granule=120000ms, delay=135000ms) 2023-05-15 14:15:17.717 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:12:48, target=2023-05-15T14:10:47, granule=120000ms, delay=121000ms) 2023-05-15 14:15:17.717 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:05:17, granule=600000ms, delay=330000ms) 2023-05-15 14:15:17.717 INFO 11 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:03:17, granule=600000ms, delay=450000ms) 2023-05-15 14:15:17.717 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=0) - not enough cached interval (source=2023-05-15T14:12:48, target=2023-05-15T14:10:47, granule=120000ms, delay=121000ms) 2023-05-15 14:15:17.718 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:10:47, target=2023-05-15T14:03:17, granule=600000ms, delay=450000ms) 2023-05-15 14:15:17.718 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=103ms, period=120000ms 2023-05-15 14:15:18.058 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:18.059 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:18.292 INFO 11 --- [pool-overview-container-5,,ae32b4e46f91] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:18.517 INFO 11 --- [pool-scanner--15978413-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:19.439 INFO 11 --- [pool-ocp-schedules-1,,cce7ed8a207b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:20.343 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:20.0, nextRun=2023-05-15 14:15:20.0, currentDate=Mon May 15 14:15:20 CST 2023 2023-05-15 14:15:20.343 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:14:20.0, nextRunTime=2023-05-15 14:15:20.0} 2023-05-15 14:15:20.347 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:15:20.348 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:20.348 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:20.524 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132031, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:20.526 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132031, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:15:20.521+08:00, endTime=null} 2023-05-15 14:15:20.550 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,016795dbcc07] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132031, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116413, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:20.521+08:00, sub_task_instance_id=132031}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:20.552 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,016795dbcc07] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:15:20.553 INFO 11 --- [ForkJoinPool-66098-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:15:20.553 INFO 11 --- [ForkJoinPool-66098-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:15:20.558 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,016795dbcc07] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132031, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:21.059 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:21.061 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:21.491 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116413, status:SUCCESS 2023-05-15 14:15:21.492 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116413, status:SUCCESS 2023-05-15 14:15:21.909 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:21.922 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:21.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:22.010 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2023-05-15 14:15:22.036 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:15:22.060 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:15:22.087 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,a123345e95cf445d,53827b480712] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=a123345e95cf445d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:22.089 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:15:22.091 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,a123345e95cf445d,53827b480712] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=a123345e95cf445d, duration=4 ms] 2023-05-15 14:15:22.108 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:15:22.127 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:15:22.151 INFO 11 --- [pool-ocp-schedules-17,,6127b79ba0f1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, count=5 2023-05-15 14:15:23.281 INFO 11 --- [pool-scanner--15978414-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:23.304 INFO 11 --- [pool-scanner--15978415-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:23.323 INFO 11 --- [pool-scanner--15978416-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:23.346 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.352 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.394 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.399 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.414 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.417 INFO 11 --- [pool-ocp-schedules-17,,e4c665b5c6fb] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:23.642 INFO 11 --- [pool-ocp-schedules-11,,c249428ad133] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:15:23.643 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:15:23.643 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:15:23.643 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:15:23.643 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:15:23.643 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:15:23.649 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:15:23.649 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:15:23.650 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:15:23.650 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:15:23.650 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:15:23.672 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:15:23.674 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:15:23.675 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:15:23.674583615+08:00 2023-05-15 14:15:23.675 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:15:23.675+08:00 2023-05-15 14:15:23.676 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:15:23.676 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:15:23.677 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:15:23.676603003+08:00 2023-05-15 14:15:23.677 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:15:23.677+08:00 2023-05-15 14:15:23.678 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:15:23.678 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:15:23.678469252+08:00 2023-05-15 14:15:23.679 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:15:23.678597629+08:00 2023-05-15 14:15:23.679 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:15:23.679+08:00 2023-05-15 14:15:23.679 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:15:23.679+08:00 2023-05-15 14:15:23.680 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:15:23.680347603+08:00 2023-05-15 14:15:23.680 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:15:23.680+08:00 2023-05-15 14:15:23.688 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:15:23.675+08:00 and status AVAILABLE 2023-05-15 14:15:23.691 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:15:23.679+08:00 and status AVAILABLE 2023-05-15 14:15:23.694 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:15:23.695 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:15:23.677+08:00 and status AVAILABLE 2023-05-15 14:15:23.697 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:15:23.714 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131323 0 0 , error= 2023-05-15 14:15:23.714 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:15:23.714+08:00 2023-05-15 14:15:23.720 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:15:23.679+08:00 and status AVAILABLE 2023-05-15 14:15:23.722 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131323 0 0 , error= 2023-05-15 14:15:23.723 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:15:23.722+08:00 2023-05-15 14:15:23.726 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:15:23.680+08:00 and status AVAILABLE 2023-05-15 14:15:23.727 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:15:23.714+08:00 and status AVAILABLE 2023-05-15 14:15:23.728 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:15:23.722+08:00 and status AVAILABLE 2023-05-15 14:15:23.728 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:15:23.730 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:15:23.730 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:15:23.747 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131323 0 0 , error= 2023-05-15 14:15:23.747 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:15:23.747+08:00 2023-05-15 14:15:23.748 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:15:23.749 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:15:23.752 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:15:23.747+08:00 and status AVAILABLE 2023-05-15 14:15:23.768 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:15:23.768 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131323 0 0 , error= 2023-05-15 14:15:23.770 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131323 0 0 , error= 2023-05-15 14:15:23.770 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:15:23.770+08:00 2023-05-15 14:15:23.770 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:15:23.770+08:00 2023-05-15 14:15:23.775 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:15:23.770+08:00 and status AVAILABLE 2023-05-15 14:15:23.775 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:15:23.770+08:00 and status AVAILABLE 2023-05-15 14:15:23.776 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:15:23.777 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:15:23.778 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:15:23.781 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:15:23.782 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:15:23.790 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:15:23.793 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:15:23.807 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:15:23.808 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:15:23.808 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:15:23.810 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:15:23.820 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:15:23.857 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:15:23.859 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:15:23.861 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:15:23.870 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:15:23.874 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:15:23.874 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:15:23.878 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:15:23.892 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:15:23.893 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:15:23.906 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:15:23.910 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:15:23.912 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:15:23.917 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:15:23.918 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:15:23.959 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:15:23.960 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:15:23.963 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:15:23.967 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:15:23.973 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:15:23.977 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:15:23.977 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:15:23.980 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:15:23.989 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:15:24.013 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:15:24.034 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:15:24.034 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:15:24.035 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:15:24.048 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:15:24.059 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:15:24.061 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:24.062 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:24.063 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:15:24.072 INFO 11 --- [pool-Check host status6,,ed4b74502edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:15:24.082 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:15:24.085 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:15:24.087 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:15:24.090 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:15:24.096 INFO 11 --- [pool-Check host status4,,01a2ddc728da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:15:24.097 INFO 11 --- [pool-Check host status21,,a00031ef373b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:15:24.100 INFO 11 --- [pool-Check host status7,,26b2b0a71040] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:15:24.163 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:15:24.175 INFO 11 --- [pool-Check host status29,,11d83fa8d2b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:15:24.191 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:15:24.191 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:15:24.191 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:15:24.191 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:15:24.191 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:15:24.198 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:15:24.198 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:15:24.198 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:15:24.198 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:15:24.200 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:15:24.226 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:15:24.226 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:15:24.227 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:15:24.229 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:15:24.229 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status20,,5aa90912d12b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status11,,f6cbc89e81db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:15:24.247 INFO 11 --- [pool-Check host status23,,23682ba10cda] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:15:24.249 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:15:24.250 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:15:24.250 INFO 11 --- [pool-Check host status27,,881e9f63262e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:15:24.252 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:15:24.252 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:15:24.252 INFO 11 --- [pool-Check host status5,,876679608c86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:15:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:24.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:24.831539+08:00, lastProcessTime=2023-05-15T14:15:14.830451+08:00 2023-05-15 14:15:24.862 INFO 11 --- [alarm-detect-9,,e94856cc6d63] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131304 ~ 1684131314 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131310=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:24.869 INFO 11 --- [alarm-detect-11,,75cfd25c4d4e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131254 ~ 1684131314 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389697.988, values={1684131300=389697.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:15:24.870 INFO 11 --- [alarm-detect-11,,75cfd25c4d4e] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131254 ~ 1684131314 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994495.358, values={1684131300=994495.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:24.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:24.831539+08:00 2023-05-15 14:15:25.505 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:25.0, nextRun=2023-05-15 14:15:25.0, currentDate=Mon May 15 14:15:25 CST 2023 2023-05-15 14:15:25.505 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:14:25.0, nextRunTime=2023-05-15 14:15:25.0} 2023-05-15 14:15:25.512 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:15:25.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:25.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:25.599 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132032, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:25.600 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132032, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:15:25.597+08:00, endTime=null} 2023-05-15 14:15:25.617 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132032, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116414, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:25.597+08:00, sub_task_instance_id=132032}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:15:25.662 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:15:25.663 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:15:25.664 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:15:25.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:15:25.665 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:15:25.671 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5c0766a8f749] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132032, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:25.853 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,92e50bc004fe4093,bf8a9da02704] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=92e50bc004fe4093, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:25.859 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,92e50bc004fe4093,bf8a9da02704] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=92e50bc004fe4093, duration=6 ms] 2023-05-15 14:15:26.230 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:26.233 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:26.247 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:26.270 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:26.598 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116414, status:SUCCESS 2023-05-15 14:15:26.600 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116414, status:SUCCESS 2023-05-15 14:15:27.062 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:27.063 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:27.172 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:27.527 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:27.540 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,7af3994469df4898,b03e99d9f88c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=7af3994469df4898, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:27.545 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,7af3994469df4898,b03e99d9f88c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=7af3994469df4898, duration=5 ms] 2023-05-15 14:15:28.276 INFO 11 --- [pool-overview-container-2,,6e761ee3fb1a] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:28.524 INFO 11 --- [pool-scanner--15978514-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:29.458 INFO 11 --- [pool-ocp-schedules-12,,814f95bc6d96] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:30.064 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:30.066 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:30.555 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.596 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.603 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.615 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.622 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.628 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.629 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.631 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.633 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.638 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.643 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.648 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.652 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.654 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.655 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.657 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.658 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.660 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.661 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.663 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.664 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.666 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.668 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.670 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.672 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:09:30.42, end=2023-05-15T14:14:30.552} 2023-05-15 14:15:30.672 INFO 11 --- [pool-13-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2023-05-15 14:15:30.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:31.0, nextRun=2023-05-15 14:15:30.0, currentDate=Mon May 15 14:15:30 CST 2023 2023-05-15 14:15:30.687 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:14:31.0, nextRunTime=2023-05-15 14:15:30.0} 2023-05-15 14:15:30.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:15:30.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:30.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:30.694 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:30.705 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:01.0, nextRun=2023-05-15 14:15:30.0, currentDate=Mon May 15 14:15:30 CST 2023 2023-05-15 14:15:30.705 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:15:01.0, nextRunTime=2023-05-15 14:15:30.0} 2023-05-15 14:15:30.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:15:30.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:15:30.709 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:15:31.708 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132034, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:31.711 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132034, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:15:31.706+08:00, endTime=null} 2023-05-15 14:15:31.726 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132035, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:31.727 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132035, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:15:31.722+08:00, endTime=null} 2023-05-15 14:15:31.738 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132034, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116415, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:31.706+08:00, sub_task_instance_id=132034}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:15:31.740 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132033, operation:EXECUTE, state: RUNNING 2023-05-15 14:15:31.743 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132033, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:15:31.737+08:00, endTime=null} 2023-05-15 14:15:31.747 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,13495e29158c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132035, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116416, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:31.722+08:00, sub_task_instance_id=132035}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:15:31.747 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,13495e29158c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:15:31.766 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132033, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116415, task_operation=execute, latest_execution_start_time=2023-05-15T14:15:31.737+08:00, sub_task_instance_id=132033}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:15:31.767 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:15:31.768 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:15:31.768 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:15:31.768 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:15:31.768 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:15:31.768 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:15:31.782 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.782 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.782 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.782 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.782 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.783 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.784 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.784 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.784 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.784 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.784 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.788 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.789 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.789 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.789 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.791 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.791 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.791 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:15:31.791 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.792 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.792 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.793 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.794 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:15:31.795 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.795 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.796 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:15:31.797 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.797 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.798 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:15:31.799 INFO 11 --- [pool-3248052-thread-4,fb27d19334114ed7,1000084959df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:15:31.799 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.799 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.800 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:15:31.800 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:15:31.800 INFO 11 --- [pool-3248052-thread-6,fb27d19334114ed7,f5ccc725ff76] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:15:31.801 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.801 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.801 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-3,fb27d19334114ed7,79c6d82fd977] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:15:31.802 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:15:31.803 INFO 11 --- [pool-3248052-thread-5,fb27d19334114ed7,78812a7b65e8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:15:31.803 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:15:31.803 INFO 11 --- [pool-3248052-thread-1,fb27d19334114ed7,2647a4cf04fd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:15:31.804 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:15:31.804 INFO 11 --- [pool-3248052-thread-2,fb27d19334114ed7,6d8746cbe849] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:15:31.805 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,13495e29158c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:15:31.808 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.808 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,13495e29158c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132035, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:31.808 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.808 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.809 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.811 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.812 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.812 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.812 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.818 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.818 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.819 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:15:31.822 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.822 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.822 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:15:31.826 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.827 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.827 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:31.831 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.831 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.831 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:31.848 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:31.857 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.857 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.858 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:31.860 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.860 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.861 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:31.867 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.867 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.867 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:31.874 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.874 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.874 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:31.885 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.885 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.886 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:15:31.887 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:15:31.903 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.903 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.903 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:15:31.913 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.913 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.914 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:15:31.919 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.920 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.921 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.921 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.922 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.924 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.924 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.924 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:15:31.933 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:15:31.936 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.936 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.937 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:31.950 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.951 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.951 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.951 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.951 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.952 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.952 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.953 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:31.955 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.956 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.957 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.957 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.958 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.962 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.962 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.963 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:31.963 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.964 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.964 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.964 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.965 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.966 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.966 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.966 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:31.985 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.985 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.985 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.986 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.986 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.987 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:31.987 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.987 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:31.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:31.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:31.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:31.999 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:31.999 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.000 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.000 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.000 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.001 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.002 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.002 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:32.002 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:32.016 INFO 11 --- [pool-ocp-schedules-19,,105ae915f075] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:32.016 INFO 11 --- [pool-ocp-schedules-19,,105ae915f075] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:32.021 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.022 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.022 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.022 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.023 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.026 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.026 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.027 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.027 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.028 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:32.028 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.028 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.029 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:32.029 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:15:32.041 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.041 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:15:32.042 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.042 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.042 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.042 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.044 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.044 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.044 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:32.068 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.068 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.069 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.069 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.069 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.071 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.071 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.071 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:15:32.073 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.074 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.075 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.075 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.076 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.083 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.083 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.083 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.083 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.084 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.084 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.084 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:32.084 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.085 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.085 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.085 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:15:32.086 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:15:32.089 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:15:32.094 INFO 11 --- [pool-ocp-schedules-4,,0f8c8eda54ca] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:15:32.096 INFO 11 --- [pool-ocp-schedules-4,,0f8c8eda54ca] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:15:32.101 INFO 11 --- [pool-ocp-schedules-14,,db156bcd063f] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:15:32.102 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.102 INFO 11 --- [pool-ocp-schedules-14,,db156bcd063f] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:15:32.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:15:32.105 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.106 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.106 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.106 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.107 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.107 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:32.109 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.110 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.110 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:15:32.118 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.118 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.118 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.118 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.119 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.119 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.119 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.119 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.120 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.120 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.120 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.120 INFO 11 --- [pool-ocp-schedules-5,,ffdad20602ea] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:15:32.120 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.120 INFO 11 --- [ForkJoinPool-66099-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:15:32.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:15:32.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:15:32.122 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:15:32.122 INFO 11 --- [pool-ocp-schedules-5,,ffdad20602ea] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:15:32.123 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.123 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.124 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.124 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.126 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:15:32.127 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,090789eaeb3c] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132034, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:32.127 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.127 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.127 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:15:32.128 INFO 11 --- [ForkJoinPool-66099-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:15:32.142 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:15:32.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.146 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:15:32.150 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.150 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.150 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.150 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.159 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.159 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.159 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:15:32.161 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.162 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:15:32.163 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.163 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.163 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:15:32.165 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.166 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.166 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.166 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.171 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.171 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.171 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:15:32.172 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:15:32.174 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:15:32.174 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:15:32.174 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:15:32.183 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,6be85711512e] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132033, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:15:32.196 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.197 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.197 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.197 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.198 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.199 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.199 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.199 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:15:32.201 WARN 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:15:32.201 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.201 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.201 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:15:32.203 INFO 11 --- [ForkJoinPool-66101-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:15:32.203 INFO 11 --- [ForkJoinPool-66101-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:15:32.203 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:15:32.220 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.220 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:15:32.239 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.246 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.247 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.247 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.247 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.248 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.251 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.252 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.253 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:15:32.254 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:15:32.254 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.254 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.257 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.257 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.257 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:15:32.260 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:15:32.265 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:15:32.265 INFO 11 --- [ForkJoinPool-66101-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130736319000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:15:32.275 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.275 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:15:32.298 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.298 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:15:32.310 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.310 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:15:32.324 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.324 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:15:32.336 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.336 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:15:32.349 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:15:32.362 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:15:32.363 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.363 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.363 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.364 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.368 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:15:32.368 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:15:32.369 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:15:32.384 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:15:32.384 INFO 11 --- [ForkJoinPool-66101-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684130746904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:15:32.752 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116415, status:SUCCESS 2023-05-15 14:15:32.754 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116415, status:SUCCESS 2023-05-15 14:15:32.764 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116416, status:SUCCESS 2023-05-15 14:15:32.765 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116416, status:SUCCESS 2023-05-15 14:15:33.066 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:33.067 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:34.827 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:34.826519+08:00, lastProcessTime=2023-05-15T14:15:24.831539+08:00 2023-05-15 14:15:34.887 INFO 11 --- [alarm-detect-1,,f15d9ceb6b98] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131314 ~ 1684131324 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131320=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:34.895 INFO 11 --- [alarm-detect-9,,2b723bc3df78] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131264 ~ 1684131324 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389697.938, values={1684131300=389697.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:15:34.896 INFO 11 --- [alarm-detect-9,,2b723bc3df78] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131264 ~ 1684131324 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994495.107, values={1684131300=994495.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:34.903 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:34.826519+08:00 2023-05-15 14:15:36.067 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:36.068 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:36.233 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:36.235 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:36.249 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:36.272 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:36.319 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,d6111077032d43a0,b8687dd52886] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=ocpcluster&version=2&ObClusterId=1, client=10.xxx.xx.10, traceId=d6111077032d43a0, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=ocpcluster,1,2,{"ObRegion":"ocpcluster","ObCluster":"ocpcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684131336316428,"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 14:15:36.323 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,d6111077032d43a0,b8687dd52886] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=ocpcluster, clusterId=1, affectRows=1 2023-05-15 14:15:36.324 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,d6111077032d43a0,b8687dd52886] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.10, traceId=d6111077032d43a0, duration=6 ms] 2023-05-15 14:15:37.175 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:38.276 INFO 11 --- [pool-overview-container-7,,b84089ba201e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:38.524 INFO 11 --- [pool-scanner--15978625-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:39.069 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:39.070 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:39.480 INFO 11 --- [pool-ocp-schedules-16,,0f131a19d1d6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:40.137 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e7b5d0615479450a,411fee1dbb3a] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=e7b5d0615479450a, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:15:40.139 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,e7b5d0615479450a,411fee1dbb3a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=e7b5d0615479450a, duration=3 ms] 2023-05-15 14:15:41.677 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1bf74d9da03e4ca6,fbe10cbff336] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=1bf74d9da03e4ca6, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:41.685 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,1bf74d9da03e4ca6,fbe10cbff336] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=1bf74d9da03e4ca6, duration=9 ms] 2023-05-15 14:15:42.070 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:42.071 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:42.539 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:44.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:44.832485+08:00, lastProcessTime=2023-05-15T14:15:34.826519+08:00 2023-05-15 14:15:44.865 INFO 11 --- [alarm-detect-13,,f629511090dc] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131324 ~ 1684131334 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131330=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:44.898 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:44.832485+08:00 2023-05-15 14:15:45.072 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:45.073 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:45.529 INFO 11 --- [pool-ocp-schedules-1,,f02d57a45afb] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514606205178257408 2023-05-15 14:15:45.565 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b188d982fb6a4623,338bdc43220d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=b188d982fb6a4623, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:45.571 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,b188d982fb6a4623,338bdc43220d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=b188d982fb6a4623, duration=7 ms] 2023-05-15 14:15:46.235 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:46.238 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:46.251 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:46.275 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:46.903 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,b395033831c04cf1,e8ac3b4da00e] c.o.o.server.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obcluster&version=2&ObClusterId=1, client=10.xxx.xx.11, traceId=b395033831c04cf1, method=OcpApiRtnObj com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obcluster,1,2,{"ObRegion":"obcluster","ObCluster":"obcluster","ObRegionId":1,"ObClusterId":1,"Type":"PRIMARY","timestamp":1684131346900598,"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"ReadonlyRsList":[]},] 2023-05-15 14:15:46.907 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,b395033831c04cf1,e8ac3b4da00e] c.o.o.obops.configurl.ConfigUrlService : updateRootServer, clusterName=obcluster, clusterId=1, affectRows=1 2023-05-15 14:15:46.908 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,b395033831c04cf1,e8ac3b4da00e] c.o.o.server.trace.RequestTracingAspect : API OK: [POST /services client=10.xxx.xx.11, traceId=b395033831c04cf1, duration=6 ms] 2023-05-15 14:15:47.189 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:48.073 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:48.074 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:48.276 INFO 11 --- [pool-overview-container-1,,79c2c2846bbc] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:48.524 INFO 11 --- [pool-scanner--15978725-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:49.510 INFO 11 --- [pool-ocp-schedules-5,,c51038bcd09f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:15:51.074 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:51.075 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:51.912 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:51.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:51.926 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:52.022 INFO 11 --- [pool-ocp-schedules-18,,cfa2cc623db9] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:52.022 INFO 11 --- [pool-ocp-schedules-18,,cfa2cc623db9] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:15:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,7b025ef642dc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10780,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40070} 2023-05-15 14:15:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,7b025ef642dc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:15:54.076 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:54.077 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:15:54.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:15:54.831689+08:00, lastProcessTime=2023-05-15T14:15:44.832485+08:00 2023-05-15 14:15:54.858 INFO 11 --- [alarm-detect-12,,6e2da560351f] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131334 ~ 1684131344 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131340=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:15:54.861 INFO 11 --- [alarm-detect-12,,6e2da560351f] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=tenant_active_memstore_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant:host=10.xxx.xx.13-2] 2023-05-15 14:15:54.889 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:15:54.831689+08:00 2023-05-15 14:15:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,8ab7c980d19b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:15:55.101 INFO 11 --- [pool-prometheus-second-schedule-1,,d90656f9ced1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:15:55.983 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,fcc481159a254295,f7b7e04815f9] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=fcc481159a254295, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:15:55.989 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,fcc481159a254295,f7b7e04815f9] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=fcc481159a254295, duration=6 ms] 2023-05-15 14:15:56.238 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:56.240 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:56.253 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:56.278 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:57.077 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:15:57.078 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:15:57.193 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:57.541 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:15:58.277 INFO 11 --- [pool-overview-container-5,,1ff21b3aba85] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:15:58.278 INFO 11 --- [pool-overview-container-5,,1ff21b3aba85] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:15:58.524 INFO 11 --- [pool-scanner--15978819-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:15:59.532 INFO 11 --- [pool-ocp-schedules-12,,551b3a292872] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:16:00.000 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:16:00.000 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:16:00.000 INFO 11 --- [pool-ocp-schedules-3,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131360 2023-05-15 14:16:00.052 INFO 11 --- [pool-ocp-schedules-10,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:16:00.078 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:00.079 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:00.612 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:14:01.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.613 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:14:01.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:16:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.621 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:00.631 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:01.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.631 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:15:01.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.634 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:16:00.634 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.635 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:00.639 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:01.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.639 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:15:01.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.641 INFO 11 --- [pool-ocp-schedules-6,,aa81ca50e2e5] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:16:00.652 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:16:00.652 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.652 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:00.657 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:01.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.657 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:15:01.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:16:00.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.670 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:31.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.676 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:15:31.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:16:00.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.680 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:01.0, nextRun=2023-05-15 14:16:00.0, currentDate=Mon May 15 14:16:00 CST 2023 2023-05-15 14:16:00.689 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:15:01.0, nextRunTime=2023-05-15 14:16:00.0} 2023-05-15 14:16:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:16:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:00.692 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:01.164 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132038, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.166 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132038, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:16:01.161+08:00, endTime=null} 2023-05-15 14:16:01.176 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132039, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.177 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132039, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:16:01.174+08:00, endTime=null} 2023-05-15 14:16:01.184 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f43538e614d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132038, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116418, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.161+08:00, sub_task_instance_id=132038}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.188 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132036, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.189 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132036, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:16:01.185+08:00, endTime=null} 2023-05-15 14:16:01.189 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:16:01.189 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:16:01.189 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:16:01.190 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:16:01.190 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:16:01.194 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,9f43538e614d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132038, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.194 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:16:01.194 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:16:01.194 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:16:01.194 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:16:01.194 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:16:01.195 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a886814800f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132039, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116419, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.174+08:00, sub_task_instance_id=132039}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.199 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132037, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.201 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132037, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=0, startTime=2023-05-15T14:16:01.197+08:00, endTime=null} 2023-05-15 14:16:01.211 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,129ffa510057] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132036, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116417, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.185+08:00, sub_task_instance_id=132036}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.211 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132042, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.213 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132042, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:16:01.209+08:00, endTime=null} 2023-05-15 14:16:01.215 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a886814800f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:16:01.216 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a886814800f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:16:01.218 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:16:01.218 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:16:01.219 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:16:01.219 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:16:01.220 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,8a886814800f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132039, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.221 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,b5f59dbb1fda] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132037, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116417, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.197+08:00, sub_task_instance_id=132037}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.223 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132040, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.223 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:16:01.227 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132040, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:16:01.220+08:00, endTime=null} 2023-05-15 14:16:01.231 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4139204d4613] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132042, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116422, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.209+08:00, sub_task_instance_id=132042}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.232 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,129ffa510057] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132036, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.236 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132041, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:01.236 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,b5f59dbb1fda] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132037, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.237 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132041, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:16:01.234+08:00, endTime=null} 2023-05-15 14:16:01.245 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132040, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116420, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.220+08:00, sub_task_instance_id=132040}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.249 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,4139204d4613] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132042, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.251 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a885f549f11] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132041, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116421, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:01.234+08:00, sub_task_instance_id=132041}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:01.251 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a885f549f11] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:16:01.265 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:16:01.265 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:16:01.266 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obcluster, sql count is 0. 2023-05-15 14:16:01.267 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, result is empty. 2023-05-15 14:16:01.271 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:16:01.271 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:16:01.272 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:16:01.272 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:16:01.272 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:16:01.273 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:16:01.274 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,4a11fb88cee5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132040, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.283 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.284 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.285 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.286 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:16:01.287 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:16:01.288 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-1,fb27d19334114ed7,255c5be29d6e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-3,fb27d19334114ed7,bcaa49f1368d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-2,fb27d19334114ed7,a23c9087f3df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:16:01.289 INFO 11 --- [pool-3248221-thread-4,fb27d19334114ed7,da6946684afc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:16:01.290 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.290 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.290 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.290 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:01.295 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.295 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.295 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:01.295 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:01.296 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:16:01.296 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:16:01.300 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:16:01.300 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:16:01.300 INFO 11 --- [pool-3248221-thread-5,fb27d19334114ed7,27276800f0a2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:16:01.300 INFO 11 --- [pool-3248221-thread-6,fb27d19334114ed7,e79b9a8a66ab] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:16:01.301 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a885f549f11] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:16:01.305 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,8a885f549f11] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132041, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:01.384 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116417, status:SUCCESS 2023-05-15 14:16:01.386 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116417, status:SUCCESS 2023-05-15 14:16:01.394 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116418, status:SUCCESS 2023-05-15 14:16:01.395 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116418, status:SUCCESS 2023-05-15 14:16:01.408 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116419, status:SUCCESS 2023-05-15 14:16:01.409 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116419, status:SUCCESS 2023-05-15 14:16:01.420 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116420, status:SUCCESS 2023-05-15 14:16:01.422 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116420, status:SUCCESS 2023-05-15 14:16:01.432 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116421, status:SUCCESS 2023-05-15 14:16:01.434 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116421, status:SUCCESS 2023-05-15 14:16:01.444 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116422, status:SUCCESS 2023-05-15 14:16:01.446 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116422, status:SUCCESS 2023-05-15 14:16:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,6cc0f966c030] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131420, interval=60 2023-05-15 14:16:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,93d47edae1ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131420, interval=60 2023-05-15 14:16:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,6cc0f966c030] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:16:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,6cc0f966c030] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131361, count=8, elapsed=1682447230159 2023-05-15 14:16:01.521 INFO 11 --- [pool-prometheus-minute-schedule-1,,93d47edae1ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:16:01.521 INFO 11 --- [pool-prometheus-minute-schedule-1,,93d47edae1ac] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131361, count=6, elapsed=1682447230160 2023-05-15 14:16:03.079 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:03.080 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:04.713 INFO 11 --- [pool-ocp-schedules-12,,836d03198468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:16:04.721 INFO 11 --- [pool-ocp-schedules-12,,836d03198468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:16:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:04.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:04.834005+08:00, lastProcessTime=2023-05-15T14:15:54.831689+08:00 2023-05-15 14:16:04.858 INFO 11 --- [alarm-detect-3,,a90a06ec8d69] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131344 ~ 1684131354 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131350=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:04.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:04.834005+08:00 2023-05-15 14:16:05.749 INFO 11 --- [pool-ocp-schedules-12,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2023-05-15 14:16:05.820 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:06.0, nextRun=2023-05-15 14:16:05.0, currentDate=Mon May 15 14:16:05 CST 2023 2023-05-15 14:16:05.820 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:15:06.0, nextRunTime=2023-05-15 14:16:05.0} 2023-05-15 14:16:05.824 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:16:05.824 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:05.824 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:06.080 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:06.082 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:06.240 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:06.243 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:06.256 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:06.280 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:06.329 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132043, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:06.331 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132043, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:16:06.327+08:00, endTime=null} 2023-05-15 14:16:06.354 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,752fc534c3a5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132043, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116423, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:06.326+08:00, sub_task_instance_id=132043}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:06.375 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,0972dad59450] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:16:06.375 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,2e36ffc185d4] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:16:06.375 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,1d0eb1cb7cbb] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:16:06.407 INFO 11 --- [pool-Check obproxy14,fb27d19334114ed7,2e36ffc185d4] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:16:06.413 INFO 11 --- [pool-Check obproxy13,fb27d19334114ed7,1d0eb1cb7cbb] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:16:06.415 INFO 11 --- [pool-Check obproxy15,fb27d19334114ed7,0972dad59450] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:16:06.427 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,752fc534c3a5] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132043, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:06.541 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116423, status:SUCCESS 2023-05-15 14:16:06.542 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116423, status:SUCCESS 2023-05-15 14:16:06.977 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,f1ebd62a90424370,dcd27f3c49ec] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=f1ebd62a90424370, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:06.982 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,f1ebd62a90424370,dcd27f3c49ec] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=f1ebd62a90424370, duration=6 ms] 2023-05-15 14:16:07.195 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:08.276 INFO 11 --- [pool-overview-container-2,,bbd401f5cf19] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:08.525 INFO 11 --- [pool-scanner--15978912-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:16:09.082 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:09.083 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:09.553 INFO 11 --- [pool-ocp-schedules-14,,d03bc98006aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:12.027 INFO 11 --- [pool-ocp-schedules-6,,33034bba2bea] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:12.027 INFO 11 --- [pool-ocp-schedules-6,,33034bba2bea] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:12.083 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:12.085 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:12.214 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131300, nodeCount=1000, affectRows=1000 2023-05-15 14:16:12.219 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131300, nodeCount=1000, validCount=1000 2023-05-15 14:16:12.256 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131300, nodeCount=1000, affectRows=1000 2023-05-15 14:16:12.264 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131300, nodeCount=1000, validCount=1000 2023-05-15 14:16:12.298 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131300, nodeCount=690, affectRows=690 2023-05-15 14:16:12.304 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131300, nodeCount=690, validCount=690 2023-05-15 14:16:12.544 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:14.168 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,0b939ce85b8c495c,c20d6570b6ce] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=0b939ce85b8c495c, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:16:14.171 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,0b939ce85b8c495c,c20d6570b6ce] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=0b939ce85b8c495c, duration=3 ms] 2023-05-15 14:16:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:14.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:14.833303+08:00, lastProcessTime=2023-05-15T14:16:04.834005+08:00 2023-05-15 14:16:14.856 INFO 11 --- [alarm-detect-9,,127deb6bce3b] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131354 ~ 1684131364 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131360=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:14.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:14.833303+08:00 2023-05-15 14:16:14.911 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,22f63ed066d84114,01bb450e3908] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=22f63ed066d84114, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:14.916 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,22f63ed066d84114,01bb450e3908] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=22f63ed066d84114, duration=5 ms] 2023-05-15 14:16:15.086 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:15.087 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:15.580 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,759d60b289bb4544,5dddacca7f0b] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=759d60b289bb4544, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:15.587 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,759d60b289bb4544,5dddacca7f0b] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=759d60b289bb4544, duration=7 ms] 2023-05-15 14:16:16.243 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:16.246 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:16.258 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:16.283 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:17.199 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:18.087 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:18.088 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:18.277 INFO 11 --- [pool-overview-container-7,,005c4a2c8ea9] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:18.521 INFO 11 --- [pool-scanner--15979012-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:19.582 INFO 11 --- [pool-ocp-schedules-17,,3a55b0ea1093] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:20.268 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:20.0, nextRun=2023-05-15 14:16:20.0, currentDate=Mon May 15 14:16:20 CST 2023 2023-05-15 14:16:20.268 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:15:20.0, nextRunTime=2023-05-15 14:16:20.0} 2023-05-15 14:16:20.273 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:16:20.273 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:20.273 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:20.553 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132044, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:20.555 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132044, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:16:20.549+08:00, endTime=null} 2023-05-15 14:16:20.575 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3a0c961409c4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132044, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116424, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:20.549+08:00, sub_task_instance_id=132044}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:20.576 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3a0c961409c4] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:16:20.577 INFO 11 --- [ForkJoinPool-66104-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:16:20.577 INFO 11 --- [ForkJoinPool-66104-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:16:20.584 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3a0c961409c4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132044, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:20.811 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116424, status:SUCCESS 2023-05-15 14:16:20.813 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116424, status:SUCCESS 2023-05-15 14:16:21.089 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:21.090 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:21.914 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:21.926 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:21.928 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:22.590 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,e94832175c7d4258,d6afb425f25c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=e94832175c7d4258, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:22.594 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,e94832175c7d4258,d6afb425f25c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=e94832175c7d4258, duration=5 ms] 2023-05-15 14:16:23.456 INFO 11 --- [pool-scanner--15979013-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:23.477 INFO 11 --- [pool-scanner--15979014-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:23.497 INFO 11 --- [pool-scanner--15979015-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:23.522 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.528 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.568 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.572 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.588 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.592 INFO 11 --- [pool-ocp-schedules-2,,aab92ddd833c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:23.981 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,293c664707c540e6,c78fc664b257] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=192.168.80.229, traceId=293c664707c540e6, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List), args=Page request [number: 0, size 2000, sort: id: DESC],] 2023-05-15 14:16:24.012 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,293c664707c540e6,c78fc664b257] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=192.168.80.229, traceId=293c664707c540e6, duration=35 ms] 2023-05-15 14:16:24.054 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,12a5aded29c74c70,e9fd4d48e58e] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=192.168.80.229, traceId=12a5aded29c74c70, method=PaginatedResponse com.oceanbase.ocp.server.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 2000, sort: id: ASC],TOP,TENANT,] 2023-05-15 14:16:24.065 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2fe04ce40c1944c5,216ac90ebb48] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?page=1&size=10, client=192.168.80.229, traceId=2fe04ce40c1944c5, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2023-05-15 14:16:24.068 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,12a5aded29c74c70,e9fd4d48e58e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=192.168.80.229, traceId=12a5aded29c74c70, duration=17 ms] 2023-05-15 14:16:24.091 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:24.091 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:24.093 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2fe04ce40c1944c5,216ac90ebb48] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=192.168.80.229, traceId=2fe04ce40c1944c5, duration=31 ms] 2023-05-15 14:16:24.134 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,070a34242d41495a,ceb73252a396] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2023-05-15T13%3A16%3A24%2B08%3A00&endTime=2023-05-15T14%3A16%3A24%2B08%3A00&metrics=tps&labels=app%3AOB&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=192.168.80.229, traceId=070a34242d41495a, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Long,String), args=2023-05-15T13:16:24+08:00,2023-05-15T14:16:24+08:00,tps,app,obregion,ob_cluster_id,tenant_name,0,0,5,360,app:OB,] 2023-05-15 14:16:24.139 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,070a34242d41495a,ceb73252a396] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:16:24.139 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fc2e0dd562e84f3d,755d3995487a] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2023-05-15T13%3A16%3A24%2B08%3A00&endTime=2023-05-15T14%3A16%3A24%2B08%3A00&metrics=sql_all_rt&labels=app%3AOB&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=192.168.80.229, traceId=fc2e0dd562e84f3d, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Long,String), args=2023-05-15T13:16:24+08:00,2023-05-15T14:16:24+08:00,sql_all_rt,app,obregion,ob_cluster_id,tenant_name,0,0,5,360,app:OB,] 2023-05-15 14:16:24.142 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,070a34242d41495a,ceb73252a396] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:16:24.144 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fc2e0dd562e84f3d,755d3995487a] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:16:24.147 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fc2e0dd562e84f3d,755d3995487a] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:16:24.153 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ecdae6b1be094d72,2a90fadc9e02] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2023-05-15T13%3A16%3A24%2B08%3A00&endTime=2023-05-15T14%3A16%3A24%2B08%3A00&metrics=tps_rt&labels=app%3AOB&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=192.168.80.229, traceId=ecdae6b1be094d72, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Long,String), args=2023-05-15T13:16:24+08:00,2023-05-15T14:16:24+08:00,tps_rt,app,obregion,ob_cluster_id,tenant_name,0,0,5,360,app:OB,] 2023-05-15 14:16:24.157 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ecdae6b1be094d72,2a90fadc9e02] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:16:24.160 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ecdae6b1be094d72,2a90fadc9e02] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:16:24.160 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,91bafc7d253e4ca1,5499a6165554] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2023-05-15T13%3A16%3A24%2B08%3A00&endTime=2023-05-15T14%3A16%3A24%2B08%3A00&metrics=sql_all_count&labels=app%3AOB&groupBy=app%2Cobregion%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=192.168.80.229, traceId=91bafc7d253e4ca1, method=IterableResponse com.oceanbase.ocp.server.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Long,String), args=2023-05-15T13:16:24+08:00,2023-05-15T14:16:24+08:00,sql_all_count,app,obregion,ob_cluster_id,tenant_name,0,0,5,360,app:OB,] 2023-05-15 14:16:24.163 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,91bafc7d253e4ca1,5499a6165554] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : tenant authorizedIdentifiers: [] 2023-05-15 14:16:24.167 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,91bafc7d253e4ca1,5499a6165554] c.o.o.s.o.ob.ObAuthorizedFacadeImpl : readable tenant ids:[1, 2, 3, 4, 5, 6] 2023-05-15 14:16:24.212 INFO 11 --- [pool-scanner--15979025-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.213 INFO 11 --- [pool-scanner--15979027-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.213 INFO 11 --- [pool-scanner--15979030-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.215 INFO 11 --- [pool-scanner--15979020-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.218 INFO 11 --- [pool-scanner--15979026-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.218 INFO 11 --- [pool-scanner--15979017-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.219 INFO 11 --- [pool-scanner--15979016-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.219 INFO 11 --- [pool-scanner--15979019-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.222 INFO 11 --- [pool-scanner--15979028-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.222 INFO 11 --- [pool-scanner--15979024-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.223 INFO 11 --- [pool-scanner--15979018-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.224 INFO 11 --- [pool-scanner--15979032-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.225 INFO 11 --- [pool-scanner--15979033-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.225 INFO 11 --- [pool-scanner--15979021-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.226 INFO 11 --- [pool-scanner--15979029-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.226 INFO 11 --- [pool-scanner--15979022-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.228 INFO 11 --- [pool-scanner--15979023-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.230 INFO 11 --- [pool-scanner--15979037-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.232 INFO 11 --- [pool-scanner--15979036-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.232 INFO 11 --- [pool-scanner--15979031-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.233 INFO 11 --- [pool-scanner--15979038-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.233 INFO 11 --- [pool-scanner--15979035-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.235 INFO 11 --- [pool-scanner--15979039-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.236 INFO 11 --- [pool-scanner--15979034-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=570, nodesFilledCount=114 2023-05-15 14:16:24.253 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,070a34242d41495a,ceb73252a396] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=192.168.80.229, traceId=070a34242d41495a, duration=122 ms] 2023-05-15 14:16:24.262 INFO 11 --- [pool-ocp-schedules-14,,8d29cd1ed6ac] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:16:24.262 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:16:24.262 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:16:24.262 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:16:24.262 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:16:24.262 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:16:24.267 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,ecdae6b1be094d72,2a90fadc9e02] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=192.168.80.229, traceId=ecdae6b1be094d72, duration=116 ms] 2023-05-15 14:16:24.267 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:16:24.267 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:16:24.267 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:16:24.268 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:16:24.268 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:16:24.289 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:16:24.289 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:16:24.289 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:16:24.290911588+08:00 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:16:24.291+08:00 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:16:24.291239387+08:00 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:16:24.291236793+08:00 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:16:24.291+08:00 2023-05-15 14:16:24.291 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:16:24.291+08:00 2023-05-15 14:16:24.293 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:16:24.293384037+08:00 2023-05-15 14:16:24.293 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:16:24.293+08:00 2023-05-15 14:16:24.294 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:16:24.291+08:00 and status AVAILABLE 2023-05-15 14:16:24.294 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:16:24.291+08:00 and status AVAILABLE 2023-05-15 14:16:24.294 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:16:24.291+08:00 and status AVAILABLE 2023-05-15 14:16:24.294 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:16:24.296 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:16:24.296356198+08:00 2023-05-15 14:16:24.296 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:16:24.293+08:00 and status AVAILABLE 2023-05-15 14:16:24.296 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:16:24.296+08:00 2023-05-15 14:16:24.297 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,91bafc7d253e4ca1,5499a6165554] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=192.168.80.229, traceId=91bafc7d253e4ca1, duration=141 ms] 2023-05-15 14:16:24.299 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:16:24.296+08:00 and status AVAILABLE 2023-05-15 14:16:24.299 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:16:24.299 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:16:24.299 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:16:24.301 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:16:24.326 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131384 0 0 , error= 2023-05-15 14:16:24.337 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131384 0 0 , error= 2023-05-15 14:16:24.337 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:16:24.326+08:00 2023-05-15 14:16:24.340 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131384 0 0 , error= 2023-05-15 14:16:24.342 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131384 0 0 , error= 2023-05-15 14:16:24.340 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:16:24.338+08:00 2023-05-15 14:16:24.342 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:16:24.342 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:16:24.342+08:00 2023-05-15 14:16:24.342 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:16:24.342+08:00 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131384 0 0 , error= 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:16:24.342+08:00 and status AVAILABLE 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:16:24.338+08:00 and status AVAILABLE 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:16:24.326+08:00 and status AVAILABLE 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:16:24.372+08:00 2023-05-15 14:16:24.372 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:16:24.342+08:00 and status AVAILABLE 2023-05-15 14:16:24.377 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:16:24.372+08:00 and status AVAILABLE 2023-05-15 14:16:24.378 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:16:24.378 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:16:24.378 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:16:24.378 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:16:24.380 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:16:24.391 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:16:24.391 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:16:24.391 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:16:24.391 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:16:24.392 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:16:24.415 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:16:24.415 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:16:24.417 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:16:24.419 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:16:24.419 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:16:24.421 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fc2e0dd562e84f3d,755d3995487a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=192.168.80.229, traceId=fc2e0dd562e84f3d, duration=284 ms] 2023-05-15 14:16:24.461 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:16:24.466 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:16:24.468 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:16:24.476 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:16:24.477 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:16:24.481 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:16:24.482 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:16:24.489 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:16:24.497 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:16:24.497 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:16:24.502 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:16:24.502 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:16:24.509 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:16:24.511 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:16:24.530 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:16:24.549 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:16:24.550 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:16:24.556 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:16:24.565 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:16:24.565 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:16:24.570 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:16:24.573 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:16:24.583 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:16:24.585 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:16:24.587 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:16:24.589 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:16:24.603 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:16:24.611 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:16:24.622 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:16:24.624 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:16:24.634 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:16:24.639 INFO 11 --- [pool-Check host status13,,477c827e9186] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:16:24.642 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:16:24.643 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:16:24.649 INFO 11 --- [pool-Check host status8,,dcc0aeca752e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:16:24.657 INFO 11 --- [pool-Check host status30,,9080bd0af8c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:16:24.659 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:16:24.670 INFO 11 --- [pool-Check host status14,,41c9e7148066] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:16:24.736 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:16:24.751 INFO 11 --- [pool-Check host status1,,43ea0a3a65ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:16:24.767 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:16:24.767 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:16:24.767 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:16:24.767 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:16:24.767 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:16:24.773 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:16:24.773 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:16:24.774 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:16:24.774 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:16:24.774 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:16:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:24.802 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:16:24.804 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:16:24.804 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:16:24.804 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:16:24.806 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:16:24.822 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:16:24.823 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:16:24.823 INFO 11 --- [pool-Check host status3,,4af301e5d388] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:16:24.824 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:16:24.824 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:16:24.825 INFO 11 --- [pool-Check host status2,,bf5bb7528193] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:16:24.825 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:16:24.825 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:16:24.825 INFO 11 --- [pool-Check host status28,,3c7974a5befa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:16:24.827 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:16:24.827 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:16:24.827 INFO 11 --- [pool-Check host status17,,743ed04e41c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:16:24.827 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:24.826526+08:00, lastProcessTime=2023-05-15T14:16:14.833303+08:00 2023-05-15 14:16:24.829 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:16:24.829 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:16:24.829 INFO 11 --- [pool-Check host status19,,ad2ed3d887ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:16:24.849 INFO 11 --- [alarm-detect-5,,553a23b9aca7] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131364 ~ 1684131374 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.355, values={1684131370=246.355}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:24.853 INFO 11 --- [alarm-detect-8,,b7cbe6bf81e5] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131314 ~ 1684131374 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389757.988, values={1684131360=389757.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:16:24.855 INFO 11 --- [alarm-detect-8,,b7cbe6bf81e5] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131314 ~ 1684131374 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994555.358, values={1684131360=994555.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:24.884 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:24.826526+08:00 2023-05-15 14:16:25.427 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:26.0, nextRun=2023-05-15 14:16:25.0, currentDate=Mon May 15 14:16:25 CST 2023 2023-05-15 14:16:25.427 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:15:26.0, nextRunTime=2023-05-15 14:16:25.0} 2023-05-15 14:16:25.438 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:16:25.438 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:25.438 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:25.633 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132045, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:25.635 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132045, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:16:25.630+08:00, endTime=null} 2023-05-15 14:16:25.658 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132045, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116425, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:25.630+08:00, sub_task_instance_id=132045}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:16:25.704 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:16:25.705 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:16:25.706 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:16:25.707 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:16:25.708 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:16:25.717 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1481449c067f] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132045, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:25.960 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116425, status:SUCCESS 2023-05-15 14:16:25.961 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116425, status:SUCCESS 2023-05-15 14:16:26.246 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:26.253 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:26.260 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:26.285 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:27.092 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:27.093 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:27.202 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:27.546 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:28.277 INFO 11 --- [pool-overview-container-1,,b664fe89a7c5] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:28.519 INFO 11 --- [pool-scanner--15979137-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:29.603 INFO 11 --- [pool-ocp-schedules-14,,e79b61c889aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:29.882 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,e2d22dba066b4b74,6de1598574e0] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=e2d22dba066b4b74, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:29.887 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,e2d22dba066b4b74,6de1598574e0] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=e2d22dba066b4b74, duration=5 ms] 2023-05-15 14:16:30.093 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:30.094 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:30.596 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:15:31.0, nextRun=2023-05-15 14:16:30.0, currentDate=Mon May 15 14:16:30 CST 2023 2023-05-15 14:16:30.596 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:15:31.0, nextRunTime=2023-05-15 14:16:30.0} 2023-05-15 14:16:30.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:16:30.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:30.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:30.603 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:30.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:16:30.0, currentDate=Mon May 15 14:16:30 CST 2023 2023-05-15 14:16:30.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:16:30.0} 2023-05-15 14:16:30.619 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:16:30.619 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:16:30.619 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:16:30.722 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132048, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:30.724 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132048, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:16:30.720+08:00, endTime=null} 2023-05-15 14:16:30.736 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132046, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:30.738 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132046, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:16:30.733+08:00, endTime=null} 2023-05-15 14:16:30.751 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132047, operation:EXECUTE, state: RUNNING 2023-05-15 14:16:30.752 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76ca8f700cc8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132048, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116427, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:30.720+08:00, sub_task_instance_id=132048}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:16:30.752 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76ca8f700cc8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:16:30.753 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132047, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:16:30.748+08:00, endTime=null} 2023-05-15 14:16:30.760 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132046, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116426, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:30.733+08:00, sub_task_instance_id=132046}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:16:30.778 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:16:30.779 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:16:30.784 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:16:30.784 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:16:30.785 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132047, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116426, task_operation=execute, latest_execution_start_time=2023-05-15T14:16:30.748+08:00, sub_task_instance_id=132047}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:16:30.788 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:16:30.789 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:16:30.864 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.865 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.866 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.867 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.867 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:16:30.868 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.869 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.869 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.869 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.869 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.870 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.870 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.870 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.870 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.870 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:16:30.871 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:16:30.871 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:16:30.871 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:16:30.871 INFO 11 --- [pool-3248383-thread-1,fb27d19334114ed7,0b8a050c5d20] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:16:30.871 INFO 11 --- [pool-3248383-thread-2,fb27d19334114ed7,22e493aa50dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:16:30.872 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:16:30.873 INFO 11 --- [pool-3248383-thread-3,fb27d19334114ed7,5dc71194f299] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:16:30.874 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:16:30.874 INFO 11 --- [pool-3248383-thread-4,fb27d19334114ed7,16279d37b2ff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:16:30.875 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.875 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.876 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.876 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.876 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.876 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.877 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.877 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.878 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.879 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.879 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.879 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.880 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.880 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.880 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:16:30.882 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.883 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.883 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.883 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.883 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:16:30.884 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:16:30.887 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:16:30.887 INFO 11 --- [pool-3248383-thread-6,fb27d19334114ed7,2ed2817951d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:16:30.888 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.888 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.888 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:16:30.889 INFO 11 --- [pool-3248383-thread-5,fb27d19334114ed7,4228086a6bd1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:16:30.889 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:30.889 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76ca8f700cc8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:16:30.894 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,76ca8f700cc8] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132048, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:30.899 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.899 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.900 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:30.910 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.910 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.910 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:30.911 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.912 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.912 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.912 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.917 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.917 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.918 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:16:30.919 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.919 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.920 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:30.922 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.922 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.922 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:30.928 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.928 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.929 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:16:30.935 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.935 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.936 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:30.938 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.938 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.938 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:16:30.947 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.947 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.947 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.947 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.947 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:30.948 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:16:30.957 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.957 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.957 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:16:30.959 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:30.959 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.960 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:30.966 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:16:30.973 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:16:30.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:30.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.983 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.984 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.985 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.985 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.985 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:30.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:30.994 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.995 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:30.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:30.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:30.996 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.007 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.008 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.008 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.008 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.009 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.011 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.012 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.012 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.012 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.012 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.013 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.013 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.013 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:31.014 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.014 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.015 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:31.023 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.023 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.029 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.030 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.030 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.030 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.031 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.035 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.036 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.036 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.044 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.045 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.045 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.045 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:31.055 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.055 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.055 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.055 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.056 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.057 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.057 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.058 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.060 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116427, status:SUCCESS 2023-05-15 14:16:31.061 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116427, status:SUCCESS 2023-05-15 14:16:31.071 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.072 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.073 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.073 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.077 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.078 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.078 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.078 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.078 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.078 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.079 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.079 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.079 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:31.080 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.080 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:16:31.094 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.095 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.095 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.095 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.095 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.096 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.096 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.096 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.097 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:16:31.098 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.099 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.100 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:16:31.100 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.100 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.101 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.105 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.105 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.106 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:16:31.107 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:16:31.110 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:16:31.117 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.117 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.117 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.117 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.118 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.120 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.120 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.120 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:31.132 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:16:31.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:16:31.136 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.137 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.145 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.145 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.146 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:31.152 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:16:31.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.154 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.155 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:16:31.157 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.159 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:16:31.160 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.160 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.161 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:31.163 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:16:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:16:31.173 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.178 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:16:31.179 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:16:31.179 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:16:31.179 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:16:31.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:16:31.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:16:31.181 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:16:31.184 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,39fde61077ae] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132047, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:31.185 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,3a9bfd64c4a9] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132046, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:16:31.697 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,63a91573e83b4571,8c3d1d4d589e] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=192.168.80.229, traceId=63a91573e83b4571, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List), args=Page request [number: 0, size 2000, sort: id: DESC],] 2023-05-15 14:16:31.698 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,24f7dbefcf0d47c1,e62aaf95b720] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=192.168.80.229, traceId=24f7dbefcf0d47c1, method=SuccessResponse com.oceanbase.ocp.server.controller.ProfileController.userInfo(), args=] 2023-05-15 14:16:31.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,dc1b182bfe4c4b1e,6d1b9ec230e3] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/loginKey?null, client=192.168.80.229, traceId=dc1b182bfe4c4b1e, method=SuccessResponse com.oceanbase.ocp.server.controller.InfoController.getLoginKey(), args=] 2023-05-15 14:16:31.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,dc1b182bfe4c4b1e,6d1b9ec230e3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/loginKey client=192.168.80.229, traceId=dc1b182bfe4c4b1e, duration=0 ms] 2023-05-15 14:16:31.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,0165b769070c41f1,b7ab4f6cc6f7] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=2&tenantId=6&status=RUNNING, client=192.168.80.229, traceId=0165b769070c41f1, method=PaginatedResponse com.oceanbase.ocp.server.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long), args=Page request [number: 0, size 2000, sort: id: DESC],RUNNING,MANUAL,2,6,] 2023-05-15 14:16:31.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,4655b112811248ed,02d8e091551a] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=192.168.80.229, traceId=4655b112811248ed, method=Map com.oceanbase.ocp.server.controller.InfoController.info(), args=] 2023-05-15 14:16:31.700 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,4655b112811248ed,02d8e091551a] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=192.168.80.229, traceId=4655b112811248ed, duration=1 ms] 2023-05-15 14:16:31.700 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,979d383c78ac4549,0a690f91c81f] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=192.168.80.229, traceId=979d383c78ac4549, method=SuccessResponse com.oceanbase.ocp.server.controller.PropertyController.getSystemInfo(), args=] 2023-05-15 14:16:31.700 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,979d383c78ac4549,0a690f91c81f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=192.168.80.229, traceId=979d383c78ac4549, duration=0 ms] 2023-05-15 14:16:31.700 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,d73bf9a9214a400f,17ec1d77e8d6] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/maz/state?null, client=192.168.80.229, traceId=d73bf9a9214a400f, method=SuccessResponse com.oceanbase.ocp.server.controller.MultiAzoneController.state(), args=] 2023-05-15 14:16:31.700 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,ab59f2a7aaa645d2,140f1ecee6a6] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.80.229, traceId=ab59f2a7aaa645d2, method=OffsetDateTime com.oceanbase.ocp.server.controller.InfoController.time(), args=] 2023-05-15 14:16:31.701 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,ab59f2a7aaa645d2,140f1ecee6a6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.80.229, traceId=ab59f2a7aaa645d2, duration=1 ms] 2023-05-15 14:16:31.701 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,8f0d840210ec47d9,cc1f112ae1cf] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/mc/info?null, client=192.168.80.229, traceId=8f0d840210ec47d9, method=SuccessResponse com.oceanbase.ocp.server.controller.OcpMultiClusterInfoController.info(), args=] 2023-05-15 14:16:31.701 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,8f0d840210ec47d9,cc1f112ae1cf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/mc/info client=192.168.80.229, traceId=8f0d840210ec47d9, duration=1 ms] 2023-05-15 14:16:31.701 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,24f7dbefcf0d47c1,e62aaf95b720] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=192.168.80.229, traceId=24f7dbefcf0d47c1, duration=3 ms] 2023-05-15 14:16:31.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,0165b769070c41f1,b7ab4f6cc6f7] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=192.168.80.229, traceId=0165b769070c41f1, duration=5 ms] 2023-05-15 14:16:31.702 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,d73bf9a9214a400f,17ec1d77e8d6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/maz/state client=192.168.80.229, traceId=d73bf9a9214a400f, duration=2 ms] 2023-05-15 14:16:31.703 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6?id=2&tenantId=6, client=192.168.80.229, traceId=9d7a0f7826d048e4, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantController.getTenant(Long,Long), args=2,6,] 2023-05-15 14:16:31.716 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,63a91573e83b4571,8c3d1d4d589e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=192.168.80.229, traceId=63a91573e83b4571, duration=21 ms] 2023-05-15 14:16:31.728 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,94b658edf0714b56,1c95998cd23f] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=192.168.80.229, traceId=94b658edf0714b56, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2023-05-15 14:16:31.728 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.729 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.729 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.730 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.738 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.738 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.739 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.739 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.739 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.743 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.743 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.744 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.MAX_IOPS, t2.MIN_IOPS FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2023-05-15 14:16:31.751 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.751 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.752 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.752 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.752 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.755 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,94b658edf0714b56,1c95998cd23f] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=192.168.80.229, traceId=94b658edf0714b56, duration=28 ms] 2023-05-15 14:16:31.763 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.763 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.764 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2023-05-15 14:16:31.775 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.775 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.776 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2023-05-15 14:16:31.781 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.781 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.781 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:31.803 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.804 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.804 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:16:31.835 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:31.836 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.836 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.836 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.837 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:31.847 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:31.847 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:31.848 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:16:31.871 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:16:31.871 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:16:31.887 ERROR 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:31.889 ERROR 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] com.alibaba.druid.pool.DruidDataSource : {dataSource-105} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:31.889 INFO 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] com.alibaba.druid.pool.DruidDataSource : {dataSource-105,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:16:31.889 INFO 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] com.alibaba.druid.pool.DruidDataSource : {dataSource-105} closing ... 2023-05-15 14:16:31.893 INFO 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] com.alibaba.druid.pool.DruidDataSource : {dataSource-105} closed 2023-05-15 14:16:31.893 ERROR 11 --- [obsdk-init-thread-6,9d7a0f7826d048e4,527b2518949a] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail 2023-05-15 14:16:31.894 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail 2023-05-15 14:16:31.894 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail 2023-05-15 14:16:31.903 ERROR 11 --- [Druid-ConnectionPool-Create-750881736,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573922) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573922) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:16:31.912 WARN 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.o.i.tenant.TenantServiceImpl : Can not get tenant extra info from ob. exception: com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:144) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:131) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:78) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.parameter.TenantParameterServiceImpl.listTenantParameter(TenantParameterServiceImpl.java:116) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.parameter.TenantParameterServiceImpl.getTenantParameter(TenantParameterServiceImpl.java:232) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.tenant.TenantServiceImpl.attachPerfParameters(TenantServiceImpl.java:410) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obops.internal.tenant.TenantServiceImpl.getTenantInfo(TenantServiceImpl.java:362) ~[ocp-obops-service-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.server.controller.ObTenantController.getTenant(ObTenantController.java:172) [classes!/:4.0.3-20230301] at com.oceanbase.ocp.server.controller.ObTenantController$$FastClassBySpringCGLIB$$8f5d377e.invoke() [classes!/:4.0.3-20230301] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) [spring-context-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) [spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.server.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:119) [classes!/:4.0.3-20230301] at sun.reflect.GeneratedMethodAccessor543.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) [spring-security-core-5.7.5.jar!/:5.7.5] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.23.jar!/:5.3.23] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) [spring-aop-5.3.23.jar!/:5.3.23] at com.oceanbase.ocp.server.controller.ObTenantController$$EnhancerBySpringCGLIB$$d343b503.getTenant() [classes!/:4.0.3-20230301] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.23.jar!/:5.3.23] at javax.servlet.http.HttpServlet.service(HttpServlet.java:670) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.23.jar!/:5.3.23] at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:111) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) [spring-webmvc-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at com.oceanbase.ocp.core.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:106) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.5.jar!/:5.7.5] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) [spring-session-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) [spring-session-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.6.13.jar!/:2.6.13] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.23.jar!/:5.3.23] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.23.jar!/:5.3.23] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at com.oceanbase.ocp.core.security.accesskey.ContentCacheRequestFilter.doFilter(ContentCacheRequestFilter.java:31) [ocp-core-4.0.3-20230301.jar!/:4.0.3-20230301] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.68.jar!/:na] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.68.jar!/:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_312] Caused by: com.oceanbase.ocp.obsdk.exception.ConnectorInitFailedException: [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail at com.oceanbase.ocp.obsdk.connector.ObConnectors.newObConnector(ObConnectors.java:257) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.createObConnector(ObConnectors.java:236) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.getObConnector(ObConnectors.java:102) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.getDataSource(ConnectTemplate.java:329) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.initJdbcTemplate(ConnectTemplate.java:317) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.(ConnectTemplate.java:53) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectTemplate.(ObConnectTemplate.java:30) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.accessor.ObAccessors.newObAccessor(ObAccessors.java:56) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.service.oceanbase.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:140) ~[ocp-service-4.0.3-20230301.jar!/:4.0.3-20230301] ... 153 common frames omitted Caused by: java.util.concurrent.ExecutionException: java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_312] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[na:1.8.0_312] at com.oceanbase.ocp.obsdk.connector.ObConnectors.newObConnector(ObConnectors.java:249) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] ... 161 common frames omitted Caused by: java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) ~[obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] ... 1 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=1573919) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:31.922 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,9d7a0f7826d048e4,e7a5368f5bf4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6 client=192.168.80.229, traceId=9d7a0f7826d048e4, duration=220 ms] 2023-05-15 14:16:32.021 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,c36b30ee5c404ea1,535675fdc885] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs/limitRule?obVersion=4.1.0.0, client=192.168.80.229, traceId=c36b30ee5c404ea1, method=SuccessResponse com.oceanbase.ocp.server.controller.ObUnitSpecController.getUnitSpecLimitRule(String), args=4.1.0.0,] 2023-05-15 14:16:32.022 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,c36b30ee5c404ea1,535675fdc885] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs/limitRule client=192.168.80.229, traceId=c36b30ee5c404ea1, duration=2 ms] 2023-05-15 14:16:32.022 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,ca2e28f53d1a4f00,68a19f7f4a69] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs?null, client=192.168.80.229, traceId=ca2e28f53d1a4f00, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObUnitSpecController.listUnitSpecs(Pageable,String,List,String), args=Page request [number: 0, size 2000, sort: id: ASC],] 2023-05-15 14:16:32.027 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,ca2e28f53d1a4f00,68a19f7f4a69] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs client=192.168.80.229, traceId=ca2e28f53d1a4f00, duration=7 ms] 2023-05-15 14:16:32.033 INFO 11 --- [pool-ocp-schedules-18,,1d74e90d94d3] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:32.033 INFO 11 --- [pool-ocp-schedules-18,,1d74e90d94d3] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:32.093 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116426, status:SUCCESS 2023-05-15 14:16:32.094 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116426, status:SUCCESS 2023-05-15 14:16:32.100 INFO 11 --- [pool-ocp-schedules-5,,36051d63245e] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:16:32.103 INFO 11 --- [pool-ocp-schedules-5,,36051d63245e] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:16:32.107 INFO 11 --- [pool-ocp-schedules-20,,d310651a53e9] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:16:32.111 INFO 11 --- [pool-ocp-schedules-20,,d310651a53e9] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:16:32.126 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.126 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.126 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.127 INFO 11 --- [pool-ocp-schedules-10,,f3b335e3923b] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:16:32.127 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.128 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.128 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.128 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.128 INFO 11 --- [ForkJoinPool-66105-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:16:32.129 INFO 11 --- [pool-ocp-schedules-10,,f3b335e3923b] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:16:32.131 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.132 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.132 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.133 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.138 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.138 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.139 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.142 INFO 11 --- [ForkJoinPool-66105-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:16:32.160 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.160 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.160 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.160 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.162 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.162 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.162 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:16:32.185 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.186 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.186 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.187 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.191 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.191 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.192 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:16:32.213 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:32.213 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.214 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.214 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.214 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.216 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.216 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.217 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:16:32.218 WARN 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:16:32.219 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.219 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.219 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:16:32.220 INFO 11 --- [ForkJoinPool-66107-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:16:32.220 INFO 11 --- [ForkJoinPool-66107-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:16:32.221 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:16:32.229 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.229 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:16:32.240 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.253 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:32.254 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.254 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.254 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.254 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.255 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:16:32.255 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.255 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.266 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:16:32.266 INFO 11 --- [ForkJoinPool-66107-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131336320000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:16:32.274 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:32.275 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.275 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.275 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.276 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.280 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.280 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.281 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:16:32.284 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.284 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.284 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:16:32.287 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:16:32.305 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.305 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:16:32.325 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.325 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:16:32.341 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.341 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:16:32.357 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.357 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:16:32.373 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.373 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:16:32.391 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:16:32.413 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:16:32.414 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.414 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.414 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.415 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.419 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:32.419 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:32.420 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:32.445 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:16:32.445 INFO 11 --- [ForkJoinPool-66107-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131346904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:16:32.940 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,053928477e99427b,ed005cf98428] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=053928477e99427b, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:32.946 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,053928477e99427b,ed005cf98428] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=053928477e99427b, duration=6 ms] 2023-05-15 14:16:33.094 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:33.099 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:34.828 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:34.828058+08:00, lastProcessTime=2023-05-15T14:16:24.826526+08:00 2023-05-15 14:16:34.868 INFO 11 --- [alarm-detect-9,,2ab5f512fc65] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131374 ~ 1684131384 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131380=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:34.872 INFO 11 --- [alarm-detect-10,,6d440ac2f663] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131324 ~ 1684131384 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389757.938, values={1684131360=389757.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:16:34.873 INFO 11 --- [alarm-detect-10,,6d440ac2f663] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131324 ~ 1684131384 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994555.107, values={1684131360=994555.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:34.900 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:34.828058+08:00 2023-05-15 14:16:36.099 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:36.100 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:36.248 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:36.255 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:36.262 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:36.287 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:37.206 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:38.277 INFO 11 --- [pool-overview-container-6,,47d81b9d3b25] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:38.525 INFO 11 --- [pool-scanner--15979248-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:39.100 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:39.102 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:39.626 INFO 11 --- [pool-ocp-schedules-1,,00ad27a93f97] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:41.262 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,7e4f9c703d2b4c19,9e282c813ef1] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=192.168.80.229, traceId=7e4f9c703d2b4c19, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2023-05-15 14:16:41.292 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,7e4f9c703d2b4c19,9e282c813ef1] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=192.168.80.229, traceId=7e4f9c703d2b4c19, duration=33 ms] 2023-05-15 14:16:41.374 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b4ca37f05f2b4872,b18a2dd9e115] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs?null, client=192.168.80.229, traceId=b4ca37f05f2b4872, method=PaginatedResponse com.oceanbase.ocp.server.controller.ObUnitSpecController.listUnitSpecs(Pageable,String,List,String), args=Page request [number: 0, size 2000, sort: id: ASC],] 2023-05-15 14:16:41.379 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,b4ca37f05f2b4872,b18a2dd9e115] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs client=192.168.80.229, traceId=b4ca37f05f2b4872, duration=7 ms] 2023-05-15 14:16:42.102 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:42.103 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:42.549 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:43.324 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/databases?id=2&tenantId=6, client=192.168.80.229, traceId=2e926341a95a4a50, method=IterableResponse com.oceanbase.ocp.server.controller.ObDatabaseController.listDatabases(Long,Long), args=2,6,] 2023-05-15 14:16:43.324 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/charsets?id=2&tenantId=6, client=192.168.80.229, traceId=675d6e2d3e764e2d, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listTenantCharsets(Long,Long), args=2,6,] 2023-05-15 14:16:43.360 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.361 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.361 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.362 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.362 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.362 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.363 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:43.363 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:43.367 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.368 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.368 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.368 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.368 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:16:43.369 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:16:43.392 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.392 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.393 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW CHARACTER SET 2023-05-15 14:16:43.401 INFO 11 --- [http-nio-0.0.0.0-8080-exec-64,675d6e2d3e764e2d,32732e342933] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/charsets client=192.168.80.229, traceId=675d6e2d3e764e2d, duration=78 ms] 2023-05-15 14:16:43.411 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:16:43.411 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:16:43.424 ERROR 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573925) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573925) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:43.425 ERROR 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] com.alibaba.druid.pool.DruidDataSource : {dataSource-106} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573925) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573925) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:43.426 INFO 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] com.alibaba.druid.pool.DruidDataSource : {dataSource-106,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:16:43.426 INFO 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] com.alibaba.druid.pool.DruidDataSource : {dataSource-106} closing ... 2023-05-15 14:16:43.426 INFO 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] com.alibaba.druid.pool.DruidDataSource : {dataSource-106} closed 2023-05-15 14:16:43.430 ERROR 11 --- [obsdk-init-thread-7,2e926341a95a4a50,beb7b7a4d578] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573925) Get Location Cache Fail 2023-05-15 14:16:43.430 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573925) Get Location Cache Fail 2023-05-15 14:16:43.431 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573925) Get Location Cache Fail 2023-05-15 14:16:43.431 WARN 11 --- [http-nio-0.0.0.0-8080-exec-52,2e926341a95a4a50,0ec0b9036eb6] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/databases client=192.168.80.229, traceId=2e926341a95a4a50, duration=108 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:16:43.439 ERROR 11 --- [Druid-ConnectionPool-Create-1097718822,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573928) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573928) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:16:43.472 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/charsets?id=2&tenantId=6, client=192.168.80.229, traceId=767cc6c4e3454e45, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listTenantCharsets(Long,Long), args=2,6,] 2023-05-15 14:16:43.503 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.504 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.504 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.504 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:43.509 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.509 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.510 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:16:43.522 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:43.522 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:43.522 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW CHARACTER SET 2023-05-15 14:16:43.527 INFO 11 --- [http-nio-0.0.0.0-8080-exec-6,767cc6c4e3454e45,487d3d5e7a93] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/charsets client=192.168.80.229, traceId=767cc6c4e3454e45, duration=56 ms] 2023-05-15 14:16:44.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:44.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:44.832733+08:00, lastProcessTime=2023-05-15T14:16:34.828058+08:00 2023-05-15 14:16:44.850 INFO 11 --- [alarm-detect-14,,a1169cc4a7dd] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131384 ~ 1684131394 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131390=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:44.884 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:44.832733+08:00 2023-05-15 14:16:45.103 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:45.104 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:45.539 INFO 11 --- [pool-ocp-schedules-17,,0f11a354dcd6] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514606205178257408 2023-05-15 14:16:46.251 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:46.257 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:46.265 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:46.289 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:47.209 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:47.221 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218} 2023-05-15 14:16:47.255 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.256 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test_tenant, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1,zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.258 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.258 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=meta_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1002, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.259 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.259 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=oms_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T16:48:31+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1006, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.261 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.261 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.267 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 3 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.269 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=obcluster, obClusterId=1, clusterId=2, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-09T14:01:25+08:00, primaryZone=zone1;zone2,zone3, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.271 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null), Interval{start=2023-05-15T14:12:47.169, end=2023-05-15T14:14:47.218},PlanChangePerfDegradationDetectService.Config(enabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=30.0, 10=10.0, 100=8.0, 1000=6.0}, limit=1000, cpuTimeLimitMs=1.0) 2023-05-15 14:16:47.271 INFO 11 --- [pool-12-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=monitor_tenant, clusterName=ocpcluster, obClusterId=1, clusterId=1, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2023-05-08T15:37:16+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, locked=false, readonly=false, obVersion=4.1.0.0, description=null, obTenantId=1004, sqlAuditEnabled=null), right=null) 2023-05-15 14:16:47.271 INFO 11 --- [pool-12-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2023-05-15 14:16:48.105 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:48.106 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:48.221 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fffab4ec3a0642ae,3e22966437f4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=fffab4ec3a0642ae, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:16:48.223 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,fffab4ec3a0642ae,3e22966437f4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=fffab4ec3a0642ae, duration=2 ms] 2023-05-15 14:16:48.277 INFO 11 --- [pool-overview-container-3,,8d7f723ebe81] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:48.525 INFO 11 --- [pool-scanner--15979348-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:49.127 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/sessions/stats?id=2&tenantId=6, client=192.168.80.229, traceId=6294e83e51f24d43, method=SuccessResponse com.oceanbase.ocp.server.controller.ObTenantSessionController.getSessionStats(Long,Long), args=2,6,] 2023-05-15 14:16:49.127 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/users?id=2&tenantId=6, client=192.168.80.229, traceId=f067a78941b0491f, method=IterableResponse com.oceanbase.ocp.server.controller.ObUserController.listDbUsers(Long,Long), args=2,6,] 2023-05-15 14:16:49.158 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:16:49.158 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:16:49.168 ERROR 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573931) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573931) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.169 ERROR 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] com.alibaba.druid.pool.DruidDataSource : {dataSource-107} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573931) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573931) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.169 INFO 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] com.alibaba.druid.pool.DruidDataSource : {dataSource-107,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:16:49.169 INFO 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] com.alibaba.druid.pool.DruidDataSource : {dataSource-107} closing ... 2023-05-15 14:16:49.169 INFO 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] com.alibaba.druid.pool.DruidDataSource : {dataSource-107} closed 2023-05-15 14:16:49.169 ERROR 11 --- [obsdk-init-thread-8,6294e83e51f24d43,03b77c138302] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573931) Get Location Cache Fail 2023-05-15 14:16:49.170 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573931) Get Location Cache Fail 2023-05-15 14:16:49.170 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:16:49.170 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573931) Get Location Cache Fail 2023-05-15 14:16:49.170 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:16:49.170 WARN 11 --- [http-nio-0.0.0.0-8080-exec-49,6294e83e51f24d43,afe284940575] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/sessions/stats client=192.168.80.229, traceId=6294e83e51f24d43, duration=44 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:16:49.200 ERROR 11 --- [Druid-ConnectionPool-Create-1765683364,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573934) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573934) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:16:49.203 ERROR 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=3152267) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=3152267) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.204 ERROR 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] com.alibaba.druid.pool.DruidDataSource : {dataSource-108} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=3152267) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=3152267) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.205 INFO 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] com.alibaba.druid.pool.DruidDataSource : {dataSource-108,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:16:49.205 INFO 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] com.alibaba.druid.pool.DruidDataSource : {dataSource-108} closing ... 2023-05-15 14:16:49.208 INFO 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] com.alibaba.druid.pool.DruidDataSource : {dataSource-108} closed 2023-05-15 14:16:49.208 ERROR 11 --- [obsdk-init-thread-9,f067a78941b0491f,d607446afe20] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=3152267) Get Location Cache Fail 2023-05-15 14:16:49.208 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=3152267) Get Location Cache Fail 2023-05-15 14:16:49.208 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=3152267) Get Location Cache Fail 2023-05-15 14:16:49.209 WARN 11 --- [http-nio-0.0.0.0-8080-exec-37,f067a78941b0491f,f35c0f2903ff] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/users client=192.168.80.229, traceId=f067a78941b0491f, duration=83 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:16:49.215 ERROR 11 --- [Druid-ConnectionPool-Create-491389473,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573937) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573937) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:16:49.253 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/databases?id=2&tenantId=6&dbName=, client=192.168.80.229, traceId=a9bd4b1f6f324ee7, method=IterableResponse com.oceanbase.ocp.server.controller.ObDatabaseController.listDatabases(Long,Long), args=2,6,] 2023-05-15 14:16:49.274 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:49.275 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:49.275 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:49.276 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:16:49.376 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:16:49.376 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:16:49.425 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:16:49.453 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:16:49.453 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:16:49.463 ERROR 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573940) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573940) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.465 ERROR 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] com.alibaba.druid.pool.DruidDataSource : {dataSource-109} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573940) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573940) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:16:49.465 INFO 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] com.alibaba.druid.pool.DruidDataSource : {dataSource-109,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:16:49.465 INFO 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] com.alibaba.druid.pool.DruidDataSource : {dataSource-109} closing ... 2023-05-15 14:16:49.466 INFO 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] com.alibaba.druid.pool.DruidDataSource : {dataSource-109} closed 2023-05-15 14:16:49.469 ERROR 11 --- [obsdk-init-thread-10,a9bd4b1f6f324ee7,6e6cba1ecde1] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573940) Get Location Cache Fail 2023-05-15 14:16:49.469 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573940) Get Location Cache Fail 2023-05-15 14:16:49.469 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573940) Get Location Cache Fail 2023-05-15 14:16:49.470 WARN 11 --- [http-nio-0.0.0.0-8080-exec-63,a9bd4b1f6f324ee7,693842f1ac02] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/databases client=192.168.80.229, traceId=a9bd4b1f6f324ee7, duration=219 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:16:49.475 ERROR 11 --- [Druid-ConnectionPool-Create-390835861,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573943) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573943) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:16:49.480 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,3eb23e172c664f22,a187ba6e2d34] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=3eb23e172c664f22, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:16:49.485 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,3eb23e172c664f22,a187ba6e2d34] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=3eb23e172c664f22, duration=5 ms] 2023-05-15 14:16:49.648 INFO 11 --- [pool-ocp-schedules-1,,6973a93ba7d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:16:51.106 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:51.107 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:51.917 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:51.928 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:51.930 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:52.038 INFO 11 --- [pool-ocp-schedules-19,,06548224ccf3] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:52.039 INFO 11 --- [pool-ocp-schedules-19,,06548224ccf3] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:16:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,b370f057ea2a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40079} 2023-05-15 14:16:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,b370f057ea2a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:16:54.107 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:54.108 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:16:54.832 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:16:54.830764+08:00, lastProcessTime=2023-05-15T14:16:44.832733+08:00 2023-05-15 14:16:54.854 INFO 11 --- [alarm-detect-11,,629afc6877fc] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131394 ~ 1684131404 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131400=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:16:54.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:16:54.830764+08:00 2023-05-15 14:16:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,16be0550c19c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:16:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,d6e87ce588cb] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:16:56.254 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:56.262 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:56.267 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:56.291 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:57.108 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:16:57.109 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:16:57.213 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:57.551 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:16:58.277 INFO 11 --- [pool-overview-container-4,,6cb399c40205] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:16:58.518 INFO 11 --- [pool-scanner--15979442-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:16:59.666 INFO 11 --- [pool-ocp-schedules-11,,b53e3ab34815] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:17:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:17:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:17:00.000 INFO 11 --- [pool-ocp-schedules-18,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131420 2023-05-15 14:17:00.051 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:17:00.110 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:00.111 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:00.470 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:17:00.0, currentDate=Mon May 15 14:17:00 CST 2023 2023-05-15 14:17:00.470 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:17:00.0} 2023-05-15 14:17:00.473 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:17:00.473 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:00.473 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:00.481 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:17:00.0, currentDate=Mon May 15 14:17:00 CST 2023 2023-05-15 14:17:00.481 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:17:00.0} 2023-05-15 14:17:00.495 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:17:00.496 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:00.496 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:00.502 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:17:00.0, currentDate=Mon May 15 14:17:00 CST 2023 2023-05-15 14:17:00.503 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:17:00.0} 2023-05-15 14:17:00.512 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:17:00.512 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:00.512 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:00.517 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:31.0, nextRun=2023-05-15 14:17:00.0, currentDate=Mon May 15 14:17:00 CST 2023 2023-05-15 14:17:00.517 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:16:31.0, nextRunTime=2023-05-15 14:17:00.0} 2023-05-15 14:17:00.520 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:17:00.520 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:00.520 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:00.527 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:17:00.0, currentDate=Mon May 15 14:17:00 CST 2023 2023-05-15 14:17:00.527 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:17:00.0} 2023-05-15 14:17:00.531 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:17:00.531 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:00.531 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:01.208 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132050, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:01.211 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132050, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:17:01.205+08:00, endTime=null} 2023-05-15 14:17:01.224 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132051, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:01.226 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132051, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:17:01.221+08:00, endTime=null} 2023-05-15 14:17:01.231 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eab901e049e0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132050, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116429, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:01.205+08:00, sub_task_instance_id=132050}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:01.236 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132049, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:01.238 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132049, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:17:01.234+08:00, endTime=null} 2023-05-15 14:17:01.241 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,184e85c583e0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132051, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116430, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:01.221+08:00, sub_task_instance_id=132051}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:01.247 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132052, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:01.248 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eab901e049e0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:17:01.249 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132052, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:17:01.245+08:00, endTime=null} 2023-05-15 14:17:01.249 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eab901e049e0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:17:01.252 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3670254c82d4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132049, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116428, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:01.234+08:00, sub_task_instance_id=132049}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:01.256 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,eab901e049e0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132050, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:01.257 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:17:01.258 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132053, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:01.258 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:17:01.258 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:17:01.258 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:17:01.258 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:17:01.258 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,184e85c583e0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:17:01.258 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,184e85c583e0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:17:01.260 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132053, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:17:01.255+08:00, endTime=null} 2023-05-15 14:17:01.261 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,184e85c583e0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:17:01.262 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:17:01.262 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:17:01.262 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:17:01.262 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:17:01.262 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,3670254c82d4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132049, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:01.262 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:17:01.265 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,78eb75c065ed] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132052, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116431, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:01.245+08:00, sub_task_instance_id=132052}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:01.265 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,78eb75c065ed] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:17:01.268 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,184e85c583e0] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132051, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:01.276 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c92077f3f6b2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132053, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116432, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:01.255+08:00, sub_task_instance_id=132053}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:01.281 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:17:01.281 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:17:01.281 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:17:01.281 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:17:01.282 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:17:01.282 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:17:01.282 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:17:01.282 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:17:01.287 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:17:01.288 INFO 11 --- [ForkJoinPool-11-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:17:01.289 INFO 11 --- [ForkJoinPool-11-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:17:01.296 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.297 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.297 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.297 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.297 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.297 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.297 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,c92077f3f6b2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132053, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.298 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.299 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:17:01.300 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:01.300 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.300 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:17:01.300 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-3,fb27d19334114ed7,c65b566c337c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-2,fb27d19334114ed7,49bd2479bcc4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-1,fb27d19334114ed7,e12bb4493118] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-4,fb27d19334114ed7,28d77abfebc6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:17:01.302 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.302 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:17:01.303 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.303 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.303 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.303 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.304 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.304 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:01.308 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.309 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.309 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:01.309 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:01.309 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:17:01.310 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:17:01.311 INFO 11 --- [ForkJoinPool-11-worker-2,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:17:01.313 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:17:01.313 INFO 11 --- [pool-3248552-thread-5,fb27d19334114ed7,0feeeadfcf6c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:17:01.313 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:17:01.313 INFO 11 --- [pool-3248552-thread-6,fb27d19334114ed7,a54271bc17cf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:17:01.314 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,78eb75c065ed] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:17:01.318 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,78eb75c065ed] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132052, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,30e1be0c2451] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131480, interval=60 2023-05-15 14:17:01.514 INFO 11 --- [pool-prometheus-minute-schedule-1,,8a458cb394c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131480, interval=60 2023-05-15 14:17:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,30e1be0c2451] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:17:01.520 INFO 11 --- [pool-prometheus-minute-schedule-2,,30e1be0c2451] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131421, count=8, elapsed=1682447290099 2023-05-15 14:17:01.521 INFO 11 --- [pool-prometheus-minute-schedule-1,,8a458cb394c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:17:01.521 INFO 11 --- [pool-prometheus-minute-schedule-1,,8a458cb394c8] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131421, count=6, elapsed=1682447290100 2023-05-15 14:17:01.645 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116428, status:SUCCESS 2023-05-15 14:17:01.648 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116428, status:SUCCESS 2023-05-15 14:17:01.659 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116429, status:SUCCESS 2023-05-15 14:17:01.660 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116429, status:SUCCESS 2023-05-15 14:17:01.673 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116430, status:SUCCESS 2023-05-15 14:17:01.674 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116430, status:SUCCESS 2023-05-15 14:17:01.682 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116431, status:SUCCESS 2023-05-15 14:17:01.683 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116431, status:SUCCESS 2023-05-15 14:17:01.691 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116432, status:SUCCESS 2023-05-15 14:17:01.692 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116432, status:SUCCESS 2023-05-15 14:17:02.287 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,9a0777c9ba92427d,3605b7779408] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=9a0777c9ba92427d, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:02.291 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,9a0777c9ba92427d,3605b7779408] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=9a0777c9ba92427d, duration=4 ms] 2023-05-15 14:17:03.111 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:03.112 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:03.699 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,27b77fe2197945a9,9e7d3ba001b3] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=27b77fe2197945a9, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:03.704 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,27b77fe2197945a9,9e7d3ba001b3] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=27b77fe2197945a9, duration=6 ms] 2023-05-15 14:17:03.780 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,05a18937d3ec407e,62ceaf5f6c8d] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=05a18937d3ec407e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:03.784 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,05a18937d3ec407e,62ceaf5f6c8d] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=05a18937d3ec407e, duration=5 ms] 2023-05-15 14:17:04.713 INFO 11 --- [pool-ocp-schedules-3,,a1fa0065229e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:17:04.721 INFO 11 --- [pool-ocp-schedules-3,,a1fa0065229e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:17:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:04.829 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:04.829101+08:00, lastProcessTime=2023-05-15T14:16:54.830764+08:00 2023-05-15 14:17:04.856 INFO 11 --- [alarm-detect-13,,3656026f1718] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131404 ~ 1684131414 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131410=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:04.896 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:04.829101+08:00 2023-05-15 14:17:05.679 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:06.0, nextRun=2023-05-15 14:17:05.0, currentDate=Mon May 15 14:17:05 CST 2023 2023-05-15 14:17:05.679 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:16:06.0, nextRunTime=2023-05-15 14:17:05.0} 2023-05-15 14:17:05.682 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:17:05.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:05.683 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:06.112 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:06.113 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:06.262 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:06.265 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:06.269 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:06.296 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:06.354 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132054, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:06.356 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132054, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:17:06.351+08:00, endTime=null} 2023-05-15 14:17:06.377 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a2e65ff2e612] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132054, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116433, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:06.351+08:00, sub_task_instance_id=132054}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:06.400 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,48c9e6433853] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:17:06.401 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,52c8ff5de4c8] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:17:06.402 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,0cc4e7ac34db] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:17:06.439 INFO 11 --- [pool-Check obproxy2,fb27d19334114ed7,48c9e6433853] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:17:06.444 INFO 11 --- [pool-Check obproxy1,fb27d19334114ed7,52c8ff5de4c8] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:17:06.445 INFO 11 --- [pool-Check obproxy3,fb27d19334114ed7,0cc4e7ac34db] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:17:06.456 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a2e65ff2e612] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132054, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:06.798 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116433, status:SUCCESS 2023-05-15 14:17:06.800 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116433, status:SUCCESS 2023-05-15 14:17:07.216 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:08.278 INFO 11 --- [pool-overview-container-8,,1060d7e8ba2f] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:17:08.279 INFO 11 --- [pool-overview-container-8,,1060d7e8ba2f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:08.525 INFO 11 --- [pool-scanner--15979535-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:17:09.113 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:09.114 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:09.684 INFO 11 --- [pool-ocp-schedules-14,,0b28d0bc625c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:09.952 INFO 11 --- [pool-ocp-schedules-9,,14e82f642325] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10770,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":5380,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40079} 2023-05-15 14:17:09.952 INFO 11 --- [pool-ocp-schedules-9,,14e82f642325] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:17:10.675 INFO 11 --- [pool-ocp-schedules-1,,56768baf0b9e] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=1 2023-05-15 14:17:12.043 INFO 11 --- [pool-ocp-schedules-19,,ae3ffc6294ad] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:12.044 INFO 11 --- [pool-ocp-schedules-19,,ae3ffc6294ad] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:12.115 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:12.116 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:12.363 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131360, nodeCount=1000, affectRows=1000 2023-05-15 14:17:12.371 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131360, nodeCount=1000, validCount=1000 2023-05-15 14:17:12.417 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131360, nodeCount=1000, affectRows=1000 2023-05-15 14:17:12.425 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131360, nodeCount=1000, validCount=1000 2023-05-15 14:17:12.460 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131360, nodeCount=690, affectRows=690 2023-05-15 14:17:12.464 INFO 11 --- [pool-metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131360, nodeCount=690, validCount=690 2023-05-15 14:17:12.554 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:14.827 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:14.826930+08:00, lastProcessTime=2023-05-15T14:17:04.829101+08:00 2023-05-15 14:17:14.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:14.826930+08:00 2023-05-15 14:17:15.057 INFO 11 --- [alarm-detect-10,,255ae296dabe] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131414 ~ 1684131424 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131420=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:15.116 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:15.117 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:16.265 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:16.268 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:16.270 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:16.298 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:17.219 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:17.587 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2023-05-15 14:17:17.621 INFO 11 --- [sql-rollup-daemon,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:17.621 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:17.622 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:17.622 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.622 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:17.625 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:17.625 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.625 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:17:17.653 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:17.654 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.655 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:17.660 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.661 INFO 11 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:17:17.664 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-1) 2023-05-15 14:17:17.664 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (ocpcluster:1-1) 2023-05-15 14:17:17.664 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-3) 2023-05-15 14:17:17.664 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (obcluster:1-7) 2023-05-15 14:17:17.670 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:17:17.670 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.670 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:17:17.670 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.671 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2983 2023-05-15 14:17:17.671 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:17.671 INFO 11 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:17:17.672 INFO 11 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:17:17.672 INFO 11 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2023-05-15 14:17:17.695 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=7, level=0->1, c=0) - source=2023-05-15T14:15:02, start=2023-05-15T14:10:47, end=2023-05-15T14:13:17, updates=100, granule=120000ms, delay=255000ms 2023-05-15 14:17:17.696 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:15:02, target=2023-05-15T14:13:17, granule=120000ms, delay=105000ms) 2023-05-15 14:17:17.696 INFO 11 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=7, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:13:17, target=2023-05-15T14:05:17, granule=600000ms, delay=480000ms) 2023-05-15 14:17:17.698 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=1, level=0->1, c=0) - source=2023-05-15T14:14:48, start=2023-05-15T14:10:47, end=2023-05-15T14:13:17, updates=133, granule=120000ms, delay=241000ms 2023-05-15 14:17:17.700 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=obcluster:1, server=3, level=0->1, c=0) - source=2023-05-15T14:14:48, start=2023-05-15T14:10:47, end=2023-05-15T14:13:17, updates=97, granule=120000ms, delay=241000ms 2023-05-15 14:17:17.700 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:14:48, target=2023-05-15T14:13:17, granule=120000ms, delay=91000ms) 2023-05-15 14:17:17.700 INFO 11 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=1, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:13:17, target=2023-05-15T14:03:17, granule=600000ms, delay=600000ms) 2023-05-15 14:17:17.702 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=0->1, c=1) - not enough cached interval (source=2023-05-15T14:14:48, target=2023-05-15T14:13:17, granule=120000ms, delay=91000ms) 2023-05-15 14:17:17.702 INFO 11 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=obcluster:1, server=3, level=1->2, c=0) - not enough cached interval (source=2023-05-15T14:13:17, target=2023-05-15T14:03:17, granule=600000ms, delay=600000ms) 2023-05-15 14:17:17.702 INFO 11 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=115ms, period=120000ms 2023-05-15 14:17:18.117 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:18.118 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:18.278 INFO 11 --- [pool-overview-container-5,,67708a9c3f97] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:18.525 INFO 11 --- [pool-scanner--15979635-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:19.274 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/charsets?id=2&tenantId=6, client=192.168.80.229, traceId=1ba046d452ed4b9e, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listTenantCharsets(Long,Long), args=2,6,] 2023-05-15 14:17:19.275 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/databases?id=2&tenantId=6, client=192.168.80.229, traceId=cfd0d17bc8c64767, method=IterableResponse com.oceanbase.ocp.server.controller.ObDatabaseController.listDatabases(Long,Long), args=2,6,] 2023-05-15 14:17:19.311 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,158d54107e964a5e,35aa1dead3e8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=158d54107e964a5e, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:19.317 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,158d54107e964a5e,35aa1dead3e8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=158d54107e964a5e, duration=6 ms] 2023-05-15 14:17:19.374 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.375 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.376 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.376 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.377 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.377 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.377 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:19.473 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.474 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.475 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:17:19.506 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=proxy, clusterName=obcluster, obClusterId=1, tenantName=test_tenant, username=root, address=127.0.0.1, port=2888, database=oceanbase) 2023-05-15 14:17:19.507 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2023-05-15 14:17:19.516 ERROR 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] com.alibaba.druid.pool.DruidDataSource : init datasource error, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"} java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573947) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573947) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:17:19.517 ERROR 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] com.alibaba.druid.pool.DruidDataSource : {dataSource-110} init error java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573947) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:927) ~[druid-1.2.8.jar!/:1.2.8] at com.oceanbase.ocp.obsdk.connector.impl.AbstractConnector.init(AbstractConnector.java:95) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.impl.DefaultConnector.init(DefaultConnector.java:14) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.obsdk.connector.ObConnectors.lambda$newObConnector$1(ObConnectors.java:243) [obsdk-4.0.3-20230301.jar!/:4.0.3-20230301] at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:41) ~[ocp-common-4.0.3-20230301.jar!/:4.0.3-20230301] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312] Caused by: java.sql.SQLTransientConnectionException: (conn=1573947) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 14 common frames omitted 2023-05-15 14:17:19.517 INFO 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] com.alibaba.druid.pool.DruidDataSource : {dataSource-110,ob-connector-datasource_proxy_obcluster_1_test_tenant_root_127.0.0.1_2888} inited 2023-05-15 14:17:19.518 INFO 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] com.alibaba.druid.pool.DruidDataSource : {dataSource-110} closing ... 2023-05-15 14:17:19.518 INFO 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] com.alibaba.druid.pool.DruidDataSource : {dataSource-110} closed 2023-05-15 14:17:19.518 ERROR 11 --- [obsdk-init-thread-11,cfd0d17bc8c64767,242029c730ec] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource failed. connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), error message:Could not connect to 127.0.0.1:2888 : (conn=1573947) Get Location Cache Fail 2023-05-15 14:17:19.518 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573947) Get Location Cache Fail 2023-05-15 14:17:19.518 ERROR 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.o.s.o.o.factory.ObAccessorFactory : [ObAccessorFactory] create tenant accessor failed, error message:[obsdk] init ob connector failed, connectProperties=ConnectProperties(connectionMode=proxy, address=127.0.0.1, port=2888, obsAddrList=null, username=root, tenantName=test_tenant, clusterName=obcluster, obClusterId=1, proxy=null, odpSocketConfig=OdpSocketConfig(obproxyConfigServerUrl=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba, automaticMatchWorkThread=false, workThreadNum=8, proxyMemLimited=1G, enableStrictKernelRelease=false, logDirSizeThreshold=1G, syslogLevel=INFO, enableMetaUsed=false), vpcId=1, compatibilityMode=MYSQL, database=oceanbase), cause:java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573947) Get Location Cache Fail 2023-05-15 14:17:19.524 WARN 11 --- [http-nio-0.0.0.0-8080-exec-62,cfd0d17bc8c64767,dd00a016396d] c.o.o.server.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/2/tenants/6/databases client=192.168.80.229, traceId=cfd0d17bc8c64767, duration=250 ms, errorMsg=[OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_CONNECT_FAILED, args=test_tenant] 2023-05-15 14:17:19.531 ERROR 11 --- [Druid-ConnectionPool-Create-531070601,,] com.alibaba.druid.pool.DruidDataSource : create connection SQLException, url: jdbc:oceanbase://127.0.0.1:2888/oceanbase?useUnicode=true&characterEncoding=UTF8&obProxySocket={"proxy_config":"automatic_match_work_thread=false,enable_metadb_used=false,enable_strict_kernel_release=false,log_dir_size_threshold=1G,obproxy_config_server_url=http://10.xxx.xx.10:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba,proxy_mem_limited=1G,syslog_level=INFO,work_thread_num=8"}, errorCode 4012, state HY000 java.sql.SQLTransientConnectionException: Could not connect to 127.0.0.1:2888 : (conn=1573950) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:235) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1644) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.Utils.retrieveProxy(Utils.java:1427) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.OceanBaseConnection.newConnection(OceanBaseConnection.java:306) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.Driver.connect(Driver.java:89) ~[oceanbase-client-2.4.2.jar!/:na] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1657) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1723) ~[druid-1.2.8.jar!/:1.2.8] at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2838) ~[druid-1.2.8.jar!/:1.2.8] Caused by: java.sql.SQLTransientConnectionException: (conn=1573950) Get Location Cache Fail at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:230) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:787) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:591) ~[oceanbase-client-2.4.2.jar!/:na] at com.oceanbase.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1639) ~[oceanbase-client-2.4.2.jar!/:na] ... 6 common frames omitted 2023-05-15 14:17:19.573 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:19.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.578 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.579 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:17:19.588 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.588 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.589 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW CHARACTER SET 2023-05-15 14:17:19.599 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,1ba046d452ed4b9e,b870861ffdef] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/charsets client=192.168.80.229, traceId=1ba046d452ed4b9e, duration=326 ms] 2023-05-15 14:17:19.623 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.o.server.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/charsets?id=2&tenantId=6, client=192.168.80.229, traceId=9f2eda5eed764632, method=IterableResponse com.oceanbase.ocp.server.controller.ObTenantController.listTenantCharsets(Long,Long), args=2,6,] 2023-05-15 14:17:19.648 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.649 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:19.649 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:19.649 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:19.704 INFO 11 --- [pool-ocp-schedules-15,,b2770cb6d3d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:20.189 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:20.0, nextRun=2023-05-15 14:17:20.0, currentDate=Mon May 15 14:17:20 CST 2023 2023-05-15 14:17:20.189 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:16:20.0, nextRunTime=2023-05-15 14:17:20.0} 2023-05-15 14:17:20.193 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:17:20.193 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:20.193 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:20.274 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:20.274 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:20.275 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2023-05-15 14:17:20.295 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:20.296 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:20.296 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW CHARACTER SET 2023-05-15 14:17:20.311 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,746d7e76638f46ed,e20b6af1d593] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=746d7e76638f46ed, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:20.317 INFO 11 --- [http-nio-0.0.0.0-8080-exec-65,746d7e76638f46ed,e20b6af1d593] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=746d7e76638f46ed, duration=6 ms] 2023-05-15 14:17:20.379 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,9f2eda5eed764632,c5639e344440] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/charsets client=192.168.80.229, traceId=9f2eda5eed764632, duration=758 ms] 2023-05-15 14:17:20.576 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132055, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:20.578 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132055, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:17:20.574+08:00, endTime=null} 2023-05-15 14:17:20.595 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,ea103592add7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132055, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116434, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:20.574+08:00, sub_task_instance_id=132055}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:20.597 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,ea103592add7] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:17:20.598 INFO 11 --- [ForkJoinPool-66110-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:17:20.598 INFO 11 --- [ForkJoinPool-66110-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:17:20.603 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,ea103592add7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132055, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:21.091 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116434, status:SUCCESS 2023-05-15 14:17:21.092 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116434, status:SUCCESS 2023-05-15 14:17:21.118 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:21.119 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:21.919 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:21.930 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:21.933 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:22.269 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,b68101826cd74fbf,bb963bc49c1e] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=b68101826cd74fbf, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:17:22.270 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,b68101826cd74fbf,bb963bc49c1e] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=b68101826cd74fbf, duration=1 ms] 2023-05-15 14:17:23.376 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,480c4a0154ff4301,c4ba0b4a8331] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=480c4a0154ff4301, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:23.383 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,480c4a0154ff4301,c4ba0b4a8331] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=480c4a0154ff4301, duration=7 ms] 2023-05-15 14:17:23.633 INFO 11 --- [pool-scanner--15979636-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:23.654 INFO 11 --- [pool-scanner--15979637-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:23.671 INFO 11 --- [pool-scanner--15979638-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:23.694 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:23.700 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:23.745 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:23.749 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:23.764 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:23.767 INFO 11 --- [pool-ocp-schedules-7,,0340733ae2d1] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:24.120 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:24.121 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:24.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:24.831201+08:00, lastProcessTime=2023-05-15T14:17:14.826930+08:00 2023-05-15 14:17:24.840 INFO 11 --- [pool-ocp-schedules-4,,c4fb4f54759e] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:17:24.841 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:17:24.841 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:17:24.841 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:17:24.841 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:17:24.841 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:17:24.850 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:17:24.850 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:17:24.850 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:17:24.851 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:17:24.855 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:17:24.865 INFO 11 --- [alarm-detect-5,,03b4bf3d4fa4] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131374 ~ 1684131434 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389817.988, values={1684131420=389817.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:17:24.865 INFO 11 --- [alarm-detect-4,,53e340706dcf] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131424 ~ 1684131434 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131430=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:24.865 INFO 11 --- [alarm-detect-5,,03b4bf3d4fa4] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131374 ~ 1684131434 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994615.358, values={1684131420=994615.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:24.877 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:17:24.878 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:17:24.879 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:17:24.879 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:17:24.879 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:17:24.879371508+08:00 2023-05-15 14:17:24.880 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:17:24.879+08:00 2023-05-15 14:17:24.880 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:17:24.880605741+08:00 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:17:24.881+08:00 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:17:24.880848323+08:00 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:17:24.881022799+08:00 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:17:24.881+08:00 2023-05-15 14:17:24.881 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:17:24.881+08:00 2023-05-15 14:17:24.882 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:17:24.879+08:00 and status AVAILABLE 2023-05-15 14:17:24.882 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:17:24.882519205+08:00 2023-05-15 14:17:24.883 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:17:24.883+08:00 2023-05-15 14:17:24.883 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:17:24.881+08:00 and status AVAILABLE 2023-05-15 14:17:24.883 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:17:24.881+08:00 and status AVAILABLE 2023-05-15 14:17:24.884 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:17:24.881+08:00 and status AVAILABLE 2023-05-15 14:17:24.885 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:17:24.883+08:00 and status AVAILABLE 2023-05-15 14:17:24.886 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:24.831201+08:00 2023-05-15 14:17:24.889 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:17:24.889 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:17:24.889 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:17:24.889 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:17:24.890 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:17:24.896 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131444 0 0 , error= 2023-05-15 14:17:24.901 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:17:24.896+08:00 2023-05-15 14:17:24.903 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131444 0 0 , error= 2023-05-15 14:17:24.907 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131444 0 0 , error= 2023-05-15 14:17:24.907 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:17:24.907+08:00 2023-05-15 14:17:24.907 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:17:24.907+08:00 2023-05-15 14:17:24.911 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:17:24.896+08:00 and status AVAILABLE 2023-05-15 14:17:24.911 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:17:24.907+08:00 and status AVAILABLE 2023-05-15 14:17:24.911 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:17:24.907+08:00 and status AVAILABLE 2023-05-15 14:17:24.913 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:17:24.913 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:17:24.913 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:17:24.922 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:17:24.923 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:17:24.923 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:17:24.950 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:17:24.952 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:17:24.952 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:17:25.001 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:17:25.013 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:17:25.015 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:17:25.024 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:17:25.031 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:17:25.035 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:17:25.041 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:17:25.049 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:17:25.068 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:17:25.087 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:17:25.093 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:17:25.103 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:17:25.106 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:17:25.121 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:17:25.123 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:17:25.144 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:17:25.157 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:17:25.170 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:17:25.171 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:17:25.180 INFO 11 --- [pool-Check host status22,,1e19ae343bd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:17:25.190 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:17:25.202 INFO 11 --- [pool-Check host status10,,67ffeff5cb3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:17:25.262 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:17:25.276 INFO 11 --- [pool-Check host status9,,645602126f7e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:17:25.359 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:25.0, nextRun=2023-05-15 14:17:25.0, currentDate=Mon May 15 14:17:25 CST 2023 2023-05-15 14:17:25.359 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:16:25.0, nextRunTime=2023-05-15 14:17:25.0} 2023-05-15 14:17:25.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:17:25.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:25.368 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:25.655 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132056, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:25.657 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132056, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:17:25.653+08:00, endTime=null} 2023-05-15 14:17:25.680 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132056, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116435, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:25.653+08:00, sub_task_instance_id=132056}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:17:25.731 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:17:25.732 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:17:25.733 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:17:25.733 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:17:25.734 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:17:25.740 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,e028753c43f2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132056, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:25.836 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131445 0 0 , error= 2023-05-15 14:17:25.839 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:17:25.839+08:00 2023-05-15 14:17:25.845 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:17:25.839+08:00 and status AVAILABLE 2023-05-15 14:17:25.847 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:17:25.860 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:17:25.890 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:17:25.930 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:17:25.946 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:17:25.964 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:17:26.020 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:17:26.036 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:17:26.055 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:17:26.108 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:17:26.124 INFO 11 --- [pool-Check host status26,,7a4469c26685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:17:26.189 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116435, status:SUCCESS 2023-05-15 14:17:26.191 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116435, status:SUCCESS 2023-05-15 14:17:26.268 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:26.270 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:26.273 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:26.300 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:26.694 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131446 0 0 , error= 2023-05-15 14:17:26.697 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:17:26.697+08:00 2023-05-15 14:17:26.700 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:17:26.697+08:00 and status AVAILABLE 2023-05-15 14:17:26.713 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:17:26.721 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:17:26.749 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:17:26.792 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:17:26.807 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:17:26.826 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:17:26.876 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:17:26.889 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:17:26.909 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:17:26.956 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:17:26.969 INFO 11 --- [pool-Check host status12,,62901d6c2326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:17:26.986 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:17:26.986 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:17:26.986 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:17:26.986 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:17:26.986 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:17:26.993 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:17:26.993 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:17:26.993 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:17:26.993 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:17:26.994 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:17:27.021 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:17:27.021 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:17:27.023 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:17:27.023 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:17:27.024 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:17:27.039 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:17:27.039 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:17:27.039 INFO 11 --- [pool-Check host status18,,37bf55b5fe96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status16,,271275f1d35d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:17:27.042 INFO 11 --- [pool-Check host status25,,5edf0da19721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:17:27.045 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:17:27.045 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:17:27.045 INFO 11 --- [pool-Check host status15,,bdaee8e97902] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:17:27.050 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:17:27.050 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:17:27.050 INFO 11 --- [pool-Check host status24,,43528e353a2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:17:27.121 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:27.122 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:27.222 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:27.561 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:28.278 INFO 11 --- [pool-overview-container-2,,21daccf72cf4] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:28.526 INFO 11 --- [pool-scanner--15979702-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:29.731 INFO 11 --- [pool-ocp-schedules-16,,56dbbd2006b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:30.122 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:30.123 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:30.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:31.0, nextRun=2023-05-15 14:17:30.0, currentDate=Mon May 15 14:17:30 CST 2023 2023-05-15 14:17:30.615 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:16:31.0, nextRunTime=2023-05-15 14:17:30.0} 2023-05-15 14:17:30.623 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:17:30.623 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:30.623 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:30.623 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:30.634 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:01.0, nextRun=2023-05-15 14:17:30.0, currentDate=Mon May 15 14:17:30 CST 2023 2023-05-15 14:17:30.634 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:17:01.0, nextRunTime=2023-05-15 14:17:30.0} 2023-05-15 14:17:30.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:17:30.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:17:30.638 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:17:30.735 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132058, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:30.737 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132058, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:17:30.732+08:00, endTime=null} 2023-05-15 14:17:30.749 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132059, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:30.751 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132059, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:17:30.746+08:00, endTime=null} 2023-05-15 14:17:30.761 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132058, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116436, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:30.732+08:00, sub_task_instance_id=132058}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:17:30.767 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132057, operation:EXECUTE, state: RUNNING 2023-05-15 14:17:30.772 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132057, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:17:30.763+08:00, endTime=null} 2023-05-15 14:17:30.773 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b350e55acbe2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132059, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116437, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:30.746+08:00, sub_task_instance_id=132059}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:17:30.773 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b350e55acbe2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:17:30.793 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132057, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116436, task_operation=execute, latest_execution_start_time=2023-05-15T14:17:30.763+08:00, sub_task_instance_id=132057}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:17:30.794 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:17:30.805 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.805 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.805 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.806 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.807 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.807 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.807 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.808 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.809 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.810 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.811 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:17:30.811 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:17:30.812 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:17:30.812 INFO 11 --- [pool-3248692-thread-1,fb27d19334114ed7,b40a28188bee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-3,fb27d19334114ed7,c1a280249ee1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-4,fb27d19334114ed7,4e7e83df0bf0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:17:30.813 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:17:30.814 INFO 11 --- [pool-3248692-thread-2,fb27d19334114ed7,a9b57b1580bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:17:30.814 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.814 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.814 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.815 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.815 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.815 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.815 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.820 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.820 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.820 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.820 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.821 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:17:30.821 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:17:30.824 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:17:30.824 INFO 11 --- [pool-3248692-thread-5,fb27d19334114ed7,aefde6eb2f25] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:17:30.826 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:17:30.826 INFO 11 --- [pool-3248692-thread-6,fb27d19334114ed7,1d822c328c46] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:17:30.826 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.827 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.827 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.827 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b350e55acbe2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:17:30.827 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.830 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.830 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.831 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.831 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.831 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,b350e55acbe2] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132059, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:30.831 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.831 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.832 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.832 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.832 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:17:30.832 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:17:30.836 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.837 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.837 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:30.843 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.843 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.843 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:30.850 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.850 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.851 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:30.856 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.857 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.857 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:30.866 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.866 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.866 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.866 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.867 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:30.867 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:30.876 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.876 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.877 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:30.884 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.884 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.884 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:30.886 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.886 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.886 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:17:30.895 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.895 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.896 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:17:30.897 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:17:30.904 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.904 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.904 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:17:30.913 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.913 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.913 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:17:30.922 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:17:30.925 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.926 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.926 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.926 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.927 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.931 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.931 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.932 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:30.944 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.945 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.945 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.945 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.945 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.946 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.946 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.947 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:30.952 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.953 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.954 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.954 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.955 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.959 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.959 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.959 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.960 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.960 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:30.960 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.960 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.960 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.961 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.961 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.962 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:30.983 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.983 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.984 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.984 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.984 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.985 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:30.985 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.985 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:30.989 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:30.990 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.990 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.990 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.991 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:30.995 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:30.995 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:30.995 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:31.000 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.000 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.000 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.000 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.001 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.002 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.002 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.002 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:31.018 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.019 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.020 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.020 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.020 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.024 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.024 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.024 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.025 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.025 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.025 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.025 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:31.025 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.026 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:17:31.026 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.026 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.027 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:31.029 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:17:31.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.037 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.038 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.038 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.038 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:31.056 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.056 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.057 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.057 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.057 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.058 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.058 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.058 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.059 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.059 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:17:31.059 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.059 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.060 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.066 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.066 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.067 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:31.071 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.072 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.072 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.072 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.073 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.074 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:17:31.075 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:17:31.078 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:17:31.078 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.081 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:17:31.082 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:31.082 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.083 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:31.094 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.095 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:31.095 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.095 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.095 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.096 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:31.096 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:17:31.097 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:17:31.097 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:17:31.098 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.098 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.099 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:17:31.114 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.117 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:17:31.118 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.118 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.118 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:17:31.123 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,6501425f2112] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132058, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:31.134 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.137 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:17:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.138 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:17:31.156 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.159 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:17:31.160 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:31.160 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:31.161 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:17:31.176 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:17:31.183 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:17:31.184 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:17:31.184 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:17:31.190 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,bc09dd1b3c25] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132057, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:17:31.287 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116436, status:SUCCESS 2023-05-15 14:17:31.289 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116436, status:SUCCESS 2023-05-15 14:17:31.304 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116437, status:SUCCESS 2023-05-15 14:17:31.305 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116437, status:SUCCESS 2023-05-15 14:17:32.049 INFO 11 --- [pool-ocp-schedules-18,,bdae360528dd] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:32.049 INFO 11 --- [pool-ocp-schedules-18,,bdae360528dd] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:32.108 INFO 11 --- [pool-ocp-schedules-12,,d7b38a8d9928] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:17:32.110 INFO 11 --- [pool-ocp-schedules-12,,d7b38a8d9928] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:17:32.116 INFO 11 --- [pool-ocp-schedules-8,,a1bdd426f8d6] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:17:32.118 INFO 11 --- [pool-ocp-schedules-8,,a1bdd426f8d6] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:17:32.133 INFO 11 --- [pool-ocp-schedules-10,,8e2347f802d3] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:17:32.134 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.135 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.135 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.135 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.136 INFO 11 --- [pool-ocp-schedules-10,,8e2347f802d3] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:17:32.136 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.136 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.137 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.138 INFO 11 --- [ForkJoinPool-66111-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:17:32.141 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.142 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.142 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.143 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.147 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.147 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.148 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.152 INFO 11 --- [ForkJoinPool-66111-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:17:32.167 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.167 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.167 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.168 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.169 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.169 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.169 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:17:32.186 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.186 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.187 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.187 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.192 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.192 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.193 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:17:32.219 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:32.220 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.220 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.220 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.220 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.222 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.222 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.222 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:17:32.223 WARN 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:17:32.223 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.224 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.224 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:17:32.225 INFO 11 --- [ForkJoinPool-66113-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:17:32.225 INFO 11 --- [ForkJoinPool-66113-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:17:32.225 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:17:32.236 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.236 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:17:32.249 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.262 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:32.263 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.263 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.263 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.263 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.264 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:17:32.264 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.265 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.275 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:32.276 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.276 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.276 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.277 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.280 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:17:32.281 INFO 11 --- [ForkJoinPool-66113-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131336320000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:17:32.282 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.282 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.283 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:17:32.285 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.286 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.286 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:17:32.289 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:17:32.307 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.307 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:17:32.328 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.328 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:17:32.345 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.345 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:17:32.361 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.361 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:17:32.377 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.377 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:17:32.394 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:17:32.413 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:17:32.414 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.415 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.415 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.415 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.420 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:17:32.420 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:17:32.421 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:17:32.437 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:17:32.438 INFO 11 --- [ForkJoinPool-66113-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131346904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:17:33.123 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:33.124 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:34.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:34.829378+08:00, lastProcessTime=2023-05-15T14:17:24.831201+08:00 2023-05-15 14:17:34.864 INFO 11 --- [alarm-detect-4,,0f8a6997a1ff] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131434 ~ 1684131444 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131440=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:34.875 INFO 11 --- [alarm-detect-8,,ed003e4b57db] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131384 ~ 1684131444 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389817.938, values={1684131420=389817.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:17:34.876 INFO 11 --- [alarm-detect-8,,ed003e4b57db] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131384 ~ 1684131444 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994615.107, values={1684131420=994615.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:34.878 INFO 11 --- [alarm-detect-8,,ed003e4b57db] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:17:34.902 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:34.829378+08:00 2023-05-15 14:17:36.124 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:36.126 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:36.271 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:36.273 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:36.275 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:36.302 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:37.258 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:37.685 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,daf2802dad1740ab,c2970173b106] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=daf2802dad1740ab, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:37.691 INFO 11 --- [http-nio-0.0.0.0-8080-exec-37,daf2802dad1740ab,c2970173b106] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=daf2802dad1740ab, duration=7 ms] 2023-05-15 14:17:38.279 INFO 11 --- [pool-overview-container-7,,f0a3fae6e05e] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:17:38.280 INFO 11 --- [pool-overview-container-7,,f0a3fae6e05e] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:38.530 INFO 11 --- [pool-scanner--15979813-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:39.126 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:39.127 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:39.755 INFO 11 --- [pool-ocp-schedules-9,,a6b2e8b58e76] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:40.688 INFO 11 --- [pool-ocp-schedules-15,,e8ebd3ec897e] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:17:42.127 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:42.128 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:42.564 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:44.786 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:44.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:44.830863+08:00, lastProcessTime=2023-05-15T14:17:34.829378+08:00 2023-05-15 14:17:44.861 INFO 11 --- [alarm-detect-13,,b1d998712548] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131444 ~ 1684131454 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131450=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:44.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:44.830863+08:00 2023-05-15 14:17:45.129 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:45.130 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:45.540 INFO 11 --- [pool-ocp-schedules-20,,4197837998bc] c.o.o.m.service.SeriesIdKeyService : Sync series key and id, currentMaxSeriesId=514606205178257408 2023-05-15 14:17:46.273 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:46.276 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:46.278 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:46.304 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:47.262 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:48.130 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:48.131 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:48.278 INFO 11 --- [pool-overview-container-1,,1f486e514cb9] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:48.527 INFO 11 --- [pool-scanner--15979913-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:49.660 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,f39b6fc9e12c4dec,6d1741871701] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=f39b6fc9e12c4dec, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:49.666 INFO 11 --- [http-nio-0.0.0.0-8080-exec-59,f39b6fc9e12c4dec,6d1741871701] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=f39b6fc9e12c4dec, duration=6 ms] 2023-05-15 14:17:49.777 INFO 11 --- [pool-ocp-schedules-1,,8f973a545230] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:17:51.132 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:51.133 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:51.922 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:51.933 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:51.935 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:52.054 INFO 11 --- [pool-ocp-schedules-14,,8a0efa7b7fc4] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:52.055 INFO 11 --- [pool-ocp-schedules-14,,8a0efa7b7fc4] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:17:53.514 INFO 11 --- [pool-prometheus-cache-monitor1,,ebcec3cab036] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":10760,"minuteWriteQueue":0,"minuteReadCacheSize":40112,"secondWriteCacheSize":2690,"seriesIdKeySize":53536,"secondWriteQueue":0,"minuteWriteCacheSize":40079} 2023-05-15 14:17:53.515 INFO 11 --- [pool-prometheus-cache-monitor1,,ebcec3cab036] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_server_resource_memory_assigned_bytes=14, tcp_retrans=25, ob_unit_num=14, go_memstats_mspan_sys_bytes=13, go_memstats_stack_sys_bytes=13, exporter_export_metrics_bytes_total=24, node_filesystem_size_bytes=73, http_server_requests_seconds_count=125, ob_compaction_frozen_time=18, ob_system_event_total_waits=29, monagent_input_sql_duration_seconds_sum=48, observer_fd_count=18, ob_plan_cache_hit_total=23, processor_process_metrics_total=18, node_filesystem_readonly=73, go_memstats_frees_total=13, sql_audit_input_collect_data_jumped_total=11, io_await=53, host_available=9, go_memstats_mallocs_total=13, ob_unit_config_max_memory_bytes=30, io_util=53, ob_system_event_detail_total_timeouts=766, ob_tenant_disk_v4_data_size=23, ob_latch_sleep_total=317, ob_tenant_unit_min_memory=30, ob_memstore_total_bytes=23, process_resident_memory_bytes=13, ob_cache_size_bytes=35, ob_tenant_assigned_cpu_total=23, ob_tenant_unit_max_memory=30, input_collect_metrics_total=25, ob_tenant_server_available_disk_free=19, node_disk_io_time_weighted_seconds_total=38, node_disk_write_time_seconds_total=38, cpu_count=25, output_write_metrics_total=12, ob_cluster_sync=8, process_open_fds=13, odp_transaction_total=14, ob_system_event_detail_time_waited_us=766, process_max_fds=13, ob_server_resource_cpu_assigned_percent=14, ocp_alarm_detect_duration_ms_count=82, proxy_memory_used_bytes=27, go_memstats_heap_inuse_bytes=13, go_memstats_mcache_sys_bytes=13, ob_tenant_disk_v4_log_size=23, node_memory_memfree_bytes=24, process_exists=33, go_gc_duration_seconds=18, ob_server_resource_disk_bytes=14, ocp_alarm_detect_duration_ms_sum=82, odp_current_session=15, node_memory_cached_bytes=24, ob_table_num=18, ob_disk_free_bytes=14, node_filesystem_avail_bytes=73, ob_unit_config_max_iops=30, ob_plan_cache_memory_bytes=23, node_network_transmit_errs_total=37, go_memstats_last_gc_time_seconds=13, go_gc_duration_seconds_count=13, monitor_exporter_avaliable=54, node_disk_read_bytes_total=38, clock_diff_milliseconds=9, ob_compaction_is_suspended=18, mysql_output_write_sql_milliseconds_summary_count=20, ob_tenant_unit_log_disk_size=23, ob_server_task_max_duration_seconds=12, go_memstats_lookups_total=13, mysql_output_write_sql_count=20, log_tailer_count=13, host_agent_process_status=13, ob_server_resource_cpu=14, go_memstats_alloc_bytes=13, ob_server_num=25, node_cpu_seconds_total_native=66, ob_session_all_num=23, ob_tenant_resource_min_memory=23, ob_memstore_freeze_times=23, ob_system_event_time_waited=29, ob_tenant_resource_min_cpu=23, ob_cluster_status_check=9, monagent_plugin_execute_seconds_total=37, node_net_bandwidth_bps=31, monagent_oceanbase_db_pool_idle=5, go_memstats_heap_idle_bytes=13, go_memstats_gc_sys_bytes=13, ob_server_resource_memory_bytes=14, ob_latch_get_total=317, go_info=14, log_tailer_processed_file_count=20, ob_tenant_assigned_mem_assigned=23, node_load15=24, ob_tenant_server_required_size=26, ob_unit_config_min_iops=30, ob_tenant_unit_max_cpu=30, ob_latch_wait_seconds_total=317, node_memory_buffers_bytes=24, go_memstats_heap_released_bytes=13, node_network_receive_bytes_total=37, ob_memstore_freeze_trigger_bytes=23, ob_waitevent_wait_seconds_total=23, ob_server_stopped_duration_seconds=16, exposer_request_latencies_count=6, ob_session_active_num=23, go_memstats_next_gc_bytes=13, node_filesystem_files_free=89, ob_memtable_max_snapshot_duration_seconds=22, node_disk_read_time_seconds_total=38, ob_compaction_frozen_version=18, ob_tenant_unit_log_disk_in_use=23, process_virtual_memory_bytes=13, ob_tenant_assigned_cpu_assigned=23, ob_tenant_unit_min_cpu=30, ob_unit_config_max_cpu=30, http_request_milliseconds_summary_count=59, monagent_oceanbase_audit_timeinfo_cache_size=5, net_connectivity=8, monagent_input_sql_duration_seconds_bucket=58, ob_latch_immediate_get_total=317, node_cpu_seconds_total=32, http_request_milliseconds_summary=62, ob_latch_immediate_miss_total=317, ob_index_error_num=9, ob_tenant_server_available_disk_total=19, ob_latch_spin_get_total=317, ob_tenant_memory_hold_bytes=14, go_memstats_buck_hash_sys_bytes=13, monagent_pipeline_report_metrics_total=14, mysql_output_write_sql_milliseconds_summary_sum=20, ob_tenant_disk_total_size=19, exposer_transferred_bytes_total=6, odp_sql_request_total=48, ob_tenant_context_memory_count=219, mysql_output_write_task_size=20, node_disk_reads_completed_total=38, exporter_export_metrics_total=24, ob_sysstat=123, node_ntp_offset_seconds=25, ob_latch_miss_total=317, go_memstats_heap_sys_bytes=13, ocp_meta_db_health=4, ob_disk_total_bytes=14, go_memstats_heap_objects=13, go_memstats_mcache_inuse_bytes=13, go_memstats_other_sys_bytes=13, go_threads=13, http_request_milliseconds_summary_sum=59, http_server_requests_seconds_sum=125, ob_memstore_active_bytes=23, go_memstats_alloc_bytes_total=13, ob_tenant_assigned_mem_total=23, ob_tenant_server_partition_replica_count=26, node_memory_memtotal_bytes=24, node_network_transmit_drop_total=37, ob_system_event_detail_total_waits=766, odp_used_connections=16, exposer_request_latencies=9, ob_waitevent_wait_total=23, odp_sql_cost_total=46, ob_tenant500_memory_used_bytes=14, node_network_transmit_bytes_total=37, ob_compaction_is_error=18, node_network_receive_drop_total=37, monagent_pipeline_execute_total=22, node_filesystem_files=89, process_start_time_seconds=13, go_memstats_heap_alloc_bytes=13, ob_tenant_resource_max_cpu=23, go_gc_duration_seconds_sum=13, go_memstats_mspan_inuse_bytes=13, ob_server_resource_memory_assigned_percent=14, mysql_output_write_sql_milliseconds_summary=23, monagent_pipeline_execute_seconds_total=22, go_memstats_sys_bytes=13, node_network_receive_errs_total=37, observer_ulimit_max_fd_count=18, ob_plan_cache_access_total=23, ob_tenant_memory_used_bytes=14, ob_tenant_server_row_count=26, monagent_oceanbase_plan_cache_size=5, ob_tenant_server_data_size=26, go_memstats_gc_cpu_fraction=13, monagent_oceanbase_audit_cache_size=5, ob_tenant_context_memory_used_bytes=219, node_disk_writes_completed_total=38, node_load1=24, process_cpu_seconds_total=13, ob_tenant_resource_max_memory=23, go_memstats_stack_inuse_bytes=13, node_load5=24, monagent_input_sql_duration_seconds_count=48, node_disk_written_bytes_total=38, monagent_oceanbase_ash_cache_size=5, ob_tenant_context_memory_hold_bytes=219, go_goroutines=13, ob_compaction_last_version=18, ob_server_resource_cpu_assigned=14, exposer_request_latencies_sum=6, ob_unit_config_min_cpu=30, process_virtual_memory_max_bytes=13, monagent_oceanbase_plan_timeinfo_cache_size=5, monagent_plugin_execute_total=37, ob_tenant500_memory_hold_bytes=14, exposer_scrapes_total=6, ob_compaction_current_timestamp=18, ob_clog_max_sync_delay_seconds=9, ob_compaction_start_time=18, oceanbase_connectivity=19} 2023-05-15 14:17:54.133 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:54.134 INFO 11 --- [pool-9-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:54.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:17:54.834 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:17:54.833624+08:00, lastProcessTime=2023-05-15T14:17:44.830863+08:00 2023-05-15 14:17:54.865 INFO 11 --- [alarm-detect-6,,da3debc4268e] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131454 ~ 1684131464 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131460=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:17:54.897 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:17:54.833624+08:00 2023-05-15 14:17:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,3a5ec2e06b56] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=9, collectTaskCount=9 2023-05-15 14:17:55.101 INFO 11 --- [pool-prometheus-second-schedule-2,,7654fe6c5769] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=10, collectTaskCount=10 2023-05-15 14:17:56.276 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:56.279 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:56.281 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:56.302 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,f8e47cc087eb4a16,dc8ceb8294cf] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=f8e47cc087eb4a16, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:17:56.304 INFO 11 --- [http-nio-0.0.0.0-8080-exec-66,f8e47cc087eb4a16,dc8ceb8294cf] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=f8e47cc087eb4a16, duration=2 ms] 2023-05-15 14:17:56.308 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:57.134 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:17:57.135 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:17:57.265 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:57.276 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,d45d6b7cca7e46a1,ad47643ba52c] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=d45d6b7cca7e46a1, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:17:57.281 INFO 11 --- [http-nio-0.0.0.0-8080-exec-60,d45d6b7cca7e46a1,ad47643ba52c] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=d45d6b7cca7e46a1, duration=6 ms] 2023-05-15 14:17:57.566 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:17:58.279 INFO 11 --- [pool-overview-container-6,,d31333a5dd21] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:17:58.525 INFO 11 --- [pool-scanner--15980007-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:17:59.800 INFO 11 --- [pool-ocp-schedules-7,,c13d26beb270] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Begin to check local odp client connectable. 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.task.scheduled.OcpSchedules : Refresh OCP properties. 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-2,,] c.o.o.s.task.scheduled.OcpSchedules : Check local odp client connectable 2023-05-15 14:18:00.000 INFO 11 --- [pool-ocp-schedules-2,,] c.o.o.s.o.OdpClientMetricCollectService : Begin to collect odp client memory, timestamp = 1684131480 2023-05-15 14:18:00.018 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:18:00.018 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:18:00.018 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:18:00.018 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:18:00.024 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : new: [] 2023-05-15 14:18:00.024 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : init spring 2023-05-15 14:18:00.024 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2023-05-15 14:18:00.024 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : starting 2023-05-15 14:18:00.033 INFO 11 --- [pool-ocp-schedules-4,,] c.o.o.s.odpclient.OdpClientChecker : Success to check local odp client connectable, connect success! 2023-05-15 14:18:00.073 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:18:00.082 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:18:00.082 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:18:00.082 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:18:00.082 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:18:00.082 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:18:00.089 WARN 11 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:18:00.089 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:18:00.136 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:00.137 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:00.422 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,400457d554c144ab,0a1f221dcdc8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=400457d554c144ab, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:18:00.428 INFO 11 --- [http-nio-0.0.0.0-8080-exec-56,400457d554c144ab,0a1f221dcdc8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=400457d554c144ab, duration=6 ms] 2023-05-15 14:18:00.504 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:16:01.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.504 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=14, name=Sync root service list, templateName=SyncRsListSchedule, scheduleRule=0 */2 * * * ?, lastRunTime=2023-05-15 14:16:01.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync root service list, nodeCount=1, taskCount=2 2023-05-15 14:18:00.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.513 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync root service list, taskDefinitionId=14, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.529 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:00.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.529 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=31, name=Get agent alarms, templateName=GetAgentAlarmsSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:17:00.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Get agent alarms, nodeCount=1, taskCount=1 2023-05-15 14:18:00.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Get agent alarms, taskDefinitionId=31, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:00.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.539 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=32, name=Collect trans stat to SqlAggregateData for alarm, templateName=CollectTransSatatSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:17:00.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.553 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect trans stat to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:18:00.553 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.553 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect trans stat to SqlAggregateData for alarm, taskDefinitionId=32, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.558 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:01.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.558 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=33, name=Collect SlowSql to SqlAggregateData for alarm, templateName=CollectSlowSqlSchedule, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:17:01.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect SlowSql to SqlAggregateData for alarm, nodeCount=1, taskCount=1 2023-05-15 14:18:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.568 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect SlowSql to SqlAggregateData for alarm, taskDefinitionId=33, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.572 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:31.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.572 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:17:31.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.575 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:18:00.575 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.575 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.581 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:01.0, nextRun=2023-05-15 14:18:00.0, currentDate=Mon May 15 14:18:00 CST 2023 2023-05-15 14:18:00.581 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=38, name=Detect alarms on schedule, templateName=BackupScheduleAlarmTemplateFactory, scheduleRule=0 */1 * * * *, lastRunTime=2023-05-15 14:17:01.0, nextRunTime=2023-05-15 14:18:00.0} 2023-05-15 14:18:00.583 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Detect alarms, nodeCount=1, taskCount=1 2023-05-15 14:18:00.584 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:00.584 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Detect alarms, taskDefinitionId=38, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:00.599 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2023-05-15 14:18:01.084 WARN 11 --- [pool-ocp-schedules-1,,] ironmentPostProcessorApplicationListener : Your Okta Issuer URL is missing. You can copy your domain from the Okta Developer Console. Follow these instructions to find it: https://bit.ly/finding-okta-domain To fix this add the `okta.oauth2.issuer` property to your application environments. 2023-05-15 14:18:01.085 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.060423123S 2023-05-15 14:18:01.085 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2023-05-15 14:18:01.204 INFO 11 --- [pool-ocp-schedules-1,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2023-05-15 14:18:01.217 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132066, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.219 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132066, name=Detect alarms of backup, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.AlarmTask, seriesId=0, startTime=2023-05-15T14:18:01.207+08:00, endTime=null} 2023-05-15 14:18:01.234 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132064, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.237 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132064, name=Collect SlowSql to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.alarm.CollectSlowSqlTask, seriesId=0, startTime=2023-05-15T14:18:01.232+08:00, endTime=null} 2023-05-15 14:18:01.243 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c70e14c5d283] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132066, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116443, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.206+08:00, sub_task_instance_id=132066}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.247 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132065, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.249 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132065, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:18:01.245+08:00, endTime=null} 2023-05-15 14:18:01.256 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,5ebb809ca7fd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132064, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116441, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.231+08:00, sub_task_instance_id=132064}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.259 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132062, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.260 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132062, name=Get all agent alarms, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.GetAgentAlarmsTask, seriesId=0, startTime=2023-05-15T14:18:01.256+08:00, endTime=null} 2023-05-15 14:18:01.262 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,c70e14c5d283] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132066, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.271 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132063, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.271 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,85e50cf2cc83] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132065, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116442, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.245+08:00, sub_task_instance_id=132065}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.272 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,85e50cf2cc83] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:18:01.273 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132063, name=Collect trans stat to SqlAggregateData for alarm, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.perf.transaction.internal.alarm.CollectTransStatTask, seriesId=0, startTime=2023-05-15T14:18:01.268+08:00, endTime=null} 2023-05-15 14:18:01.275 INFO 11 --- [pool-ocp-schedules-1,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}, BootstrapPropertySource {name='bootstrapProperties-ocp-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-composite'}, BootstrapPropertySource {name='bootstrapProperties-application-default'}] 2023-05-15 14:18:01.278 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,5ebb809ca7fd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpcluster, sql count is 0. 2023-05-15 14:18:01.278 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,5ebb809ca7fd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster ocpcluster, result is empty. 2023-05-15 14:18:01.279 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e17e7283d711] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132062, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116439, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.256+08:00, sub_task_instance_id=132062}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.280 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,5ebb809ca7fd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obcluster, exclude result is empty. 2023-05-15 14:18:01.285 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132060, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:01.286 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,5ebb809ca7fd] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132064, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.287 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132060, name=Refresh root service list, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask, seriesId=1, startTime=2023-05-15T14:18:01.282+08:00, endTime=null} 2023-05-15 14:18:01.287 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2023-05-15 14:18:01.287 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2023-05-15 14:18:01.287 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2023-05-15 14:18:01.287 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2023-05-15 14:18:01.287 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2023-05-15 14:18:01.291 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6da0c3b807aa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132063, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116440, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.268+08:00, sub_task_instance_id=132063}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.291 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,e17e7283d711] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132062, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.293 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:18:01.293 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:18:01.293 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:18:01.293 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:18:01.294 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:18:01.297 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:18:01.311 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.311 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.312 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.332 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:01.332 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6da0c3b807aa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpcluster, trans stat result is empty. 2023-05-15 14:18:01.336 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.337 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.337 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.337 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6da0c3b807aa] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obcluster, trans stat result is empty. 2023-05-15 14:18:01.337 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.337 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.337 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:01.338 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:01.342 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.343 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.343 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.344 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.344 INFO 11 --- [pool-subtask-executor30,fb27d19334114ed7,6da0c3b807aa] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132063, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.344 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.344 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.344 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:01.345 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:01.347 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62889/api/v1/log/alarms, params:null 2023-05-15 14:18:01.347 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62889/api/v1/log/alarms, params:null 2023-05-15 14:18:01.349 INFO 11 --- [ForkJoinPool-11-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62889/api/v1/log/alarms, params:null 2023-05-15 14:18:01.350 INFO 11 --- [ForkJoinPool-11-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62889/api/v1/log/alarms, params:null 2023-05-15 14:18:01.350 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62889/api/v1/log/alarms, params:null 2023-05-15 14:18:01.351 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=4, alarms size 1 2023-05-15 14:18:01.351 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=3, alarms size 2 2023-05-15 14:18:01.351 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2023-05-15 14:18:01.351 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2023-05-15 14:18:01.351 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set system property ocp.version to 4.0.3 2023-05-15 14:18:01.351 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : set systemProperties as the first PropertySource 2023-05-15 14:18:01.351 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.0.3 2023-05-15 14:18:01.352 INFO 11 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : The following 2 profiles are active: "composite", "default" 2023-05-15 14:18:01.352 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1a5fe4f82cb7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132061, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116438, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:01.298+08:00, sub_task_instance_id=132061}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:18:01.353 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : get host agent alarms, hostId=2, alarms size 2 2023-05-15 14:18:01.354 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.354 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.355 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:18:01.357 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:18:01.357 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,b611140fb3f1] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132060, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.358 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.358 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.359 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:18:01.361 INFO 11 --- [ForkJoinPool-11-worker-6,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:18:01.361 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:18:01.362 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:18:01.362 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:18:01.362 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.362 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.362 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:18:01.363 INFO 11 --- [pool-3248861-thread-4,fb27d19334114ed7,a06a04f12f6e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:18:01.363 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2023-05-15 14:18:01.364 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:01.364 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.364 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:18:01.365 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:18:01.365 INFO 11 --- [pool-3248861-thread-2,fb27d19334114ed7,aa3b21be2330] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:18:01.366 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,1a5fe4f82cb7] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132061, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.367 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.367 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.367 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:18:01.367 INFO 11 --- [pool-3248861-thread-1,fb27d19334114ed7,93cb6846bc71] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:18:01.368 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:18:01.369 INFO 11 --- [ForkJoinPool-11-worker-3,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:18:01.369 INFO 11 --- [pool-3248861-thread-3,fb27d19334114ed7,201b0b16acd9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:18:01.371 INFO 11 --- [ForkJoinPool-11-worker-4,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm observer_all_error_log_alarm null 2023-05-15 14:18:01.374 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:18:01.374 INFO 11 --- [pool-3248861-thread-5,fb27d19334114ed7,ddf977395d53] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:18:01.376 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:01.376 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:01.376 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:18:01.382 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:18:01.382 INFO 11 --- [pool-3248861-thread-6,fb27d19334114ed7,a7aa0c1f8c3c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:18:01.383 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,85e50cf2cc83] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:18:01.388 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,85e50cf2cc83] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132065, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:01.513 INFO 11 --- [pool-prometheus-minute-schedule-2,,87f6c5d22490] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131540, interval=60 2023-05-15 14:18:01.515 INFO 11 --- [pool-prometheus-minute-schedule-1,,4829beda998c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1684131540, interval=60 2023-05-15 14:18:01.540 INFO 11 --- [pool-prometheus-minute-schedule-1,,4829beda998c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=6, collectTaskCount=6 2023-05-15 14:18:01.540 INFO 11 --- [pool-prometheus-minute-schedule-1,,4829beda998c] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131481, count=6, elapsed=1682447350059 2023-05-15 14:18:01.543 INFO 11 --- [pool-prometheus-minute-schedule-2,,87f6c5d22490] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2023-05-15 14:18:01.543 INFO 11 --- [pool-prometheus-minute-schedule-2,,87f6c5d22490] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1684131481, count=8, elapsed=1682447350062 2023-05-15 14:18:01.552 INFO 11 --- [pool-ocp-schedules-1,,] o.s.boot.SpringApplication : Started application in 1.534 seconds (JVM running for 601137.525) 2023-05-15 14:18:01.552 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.534612542S 2023-05-15 14:18:01.825 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] driver class=com.oceanbase.jdbc.Driver 2023-05-15 14:18:01.825 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] username=meta_user@meta_tenant#ocpcluster 2023-05-15 14:18:01.825 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.s.c.w.MetadbDataSourceWrapper : [metadb] jdbc url=jdbc:oceanbase://10.xxx.xx.10:2983/meta_db?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true 2023-05-15 14:18:01.825 INFO 11 --- [pool-ocp-schedules-1,,] c.o.o.b.spring.DataSourceInterceptor : dataSource metaDataSource already initialized. skipped. class: com.oceanbase.ocp.server.config.wrapper.MetadbDataSourceWrapper 2023-05-15 14:18:01.884 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116438, status:SUCCESS 2023-05-15 14:18:01.886 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116438, status:SUCCESS 2023-05-15 14:18:01.895 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116439, status:SUCCESS 2023-05-15 14:18:01.897 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116439, status:SUCCESS 2023-05-15 14:18:01.909 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116440, status:SUCCESS 2023-05-15 14:18:01.910 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116440, status:SUCCESS 2023-05-15 14:18:01.922 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116441, status:SUCCESS 2023-05-15 14:18:01.924 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116441, status:SUCCESS 2023-05-15 14:18:01.936 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116442, status:SUCCESS 2023-05-15 14:18:01.938 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116442, status:SUCCESS 2023-05-15 14:18:01.950 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116443, status:SUCCESS 2023-05-15 14:18:01.951 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116443, status:SUCCESS 2023-05-15 14:18:02.106 INFO 11 --- [pool-ocp-schedules-1,,] c.o.ocp.core.property.PropertyService : Refresh config properties done 2023-05-15 14:18:03.137 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:03.138 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:04.713 INFO 11 --- [pool-ocp-schedules-20,,df49b50dbc2b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2023-05-15 14:18:04.724 INFO 11 --- [pool-ocp-schedules-20,,df49b50dbc2b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2023-05-15 14:18:04.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:18:04.830 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:18:04.829668+08:00, lastProcessTime=2023-05-15T14:17:54.833624+08:00 2023-05-15 14:18:04.862 INFO 11 --- [alarm-detect-7,,ae75e89c1ff1] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131464 ~ 1684131474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131470=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:04.888 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:18:04.829668+08:00 2023-05-15 14:18:05.729 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:06.0, nextRun=2023-05-15 14:18:05.0, currentDate=Mon May 15 14:18:05 CST 2023 2023-05-15 14:18:05.729 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=4, name=Check obproxy, templateName=CheckObproxyServerSchedule, scheduleRule=5 * * * * *, lastRunTime=2023-05-15 14:17:06.0, nextRunTime=2023-05-15 14:18:05.0} 2023-05-15 14:18:05.734 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check obproxy, nodeCount=1, taskCount=1 2023-05-15 14:18:05.734 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:05.734 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check obproxy, taskDefinitionId=4, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:06.138 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:06.141 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:06.279 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:06.281 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:06.284 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:06.310 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:06.393 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132067, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:06.396 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132067, name=Check obproxy connection status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obproxy.CheckObproxyTask, seriesId=0, startTime=2023-05-15T14:18:06.384+08:00, endTime=null} 2023-05-15 14:18:06.413 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2526b8507c8d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132067, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116444, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:06.384+08:00, sub_task_instance_id=132067}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:06.428 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,c1e8ff28f985] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:18:06.428 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,6183f6401d86] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:18:06.428 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,256a44ef1754] c.o.o.s.o.o.SystemOdpOperatorFactory : Failed to obtain password from system vault, return default password 2023-05-15 14:18:06.470 INFO 11 --- [pool-Check obproxy6,fb27d19334114ed7,6183f6401d86] c.o.o.s.o.obproxy.ObproxyService : check obproxy 8 success, update last active time to now 2023-05-15 14:18:06.472 INFO 11 --- [pool-Check obproxy5,fb27d19334114ed7,c1e8ff28f985] c.o.o.s.o.obproxy.ObproxyService : check obproxy 9 success, update last active time to now 2023-05-15 14:18:06.481 INFO 11 --- [pool-Check obproxy4,fb27d19334114ed7,256a44ef1754] c.o.o.s.o.obproxy.ObproxyService : check obproxy 7 success, update last active time to now 2023-05-15 14:18:06.497 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,2526b8507c8d] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132067, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:07.048 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116444, status:SUCCESS 2023-05-15 14:18:07.049 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116444, status:SUCCESS 2023-05-15 14:18:07.268 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:07.687 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,7c949307263d4fa0,824a31eb22b8] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.12, traceId=7c949307263d4fa0, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:18:07.692 INFO 11 --- [http-nio-0.0.0.0-8080-exec-52,7c949307263d4fa0,824a31eb22b8] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.12, traceId=7c949307263d4fa0, duration=6 ms] 2023-05-15 14:18:08.281 INFO 11 --- [pool-overview-container-2,,0b5b2b8d9d1c] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:18:08.282 INFO 11 --- [pool-overview-container-2,,0b5b2b8d9d1c] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:08.527 INFO 11 --- [pool-scanner--15980100-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=12, nodesFilledCount=0 2023-05-15 14:18:09.142 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:09.143 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:09.821 INFO 11 --- [pool-ocp-schedules-17,,a9eac0e1bbf2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:18:10.705 INFO 11 --- [pool-ocp-schedules-3,,f27befb8cee9] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=5 2023-05-15 14:18:11.584 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,8b84e07eb7cb42fb,0371393f1dad] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=8b84e07eb7cb42fb, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:18:11.590 INFO 11 --- [http-nio-0.0.0.0-8080-exec-49,8b84e07eb7cb42fb,0371393f1dad] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=8b84e07eb7cb42fb, duration=6 ms] 2023-05-15 14:18:12.061 INFO 11 --- [pool-ocp-schedules-10,,40fa77c9e60c] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:18:12.061 INFO 11 --- [pool-ocp-schedules-10,,40fa77c9e60c] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:18:12.143 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:12.144 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:12.524 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131420, nodeCount=1000, affectRows=1000 2023-05-15 14:18:12.532 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131420, nodeCount=1000, validCount=1000 2023-05-15 14:18:12.569 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:12.575 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131420, nodeCount=1000, affectRows=1000 2023-05-15 14:18:12.585 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131420, nodeCount=1000, validCount=1000 2023-05-15 14:18:12.626 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1684131420, nodeCount=690, affectRows=690 2023-05-15 14:18:12.632 INFO 11 --- [pool-metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1684131420, nodeCount=690, validCount=690 2023-05-15 14:18:14.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:18:14.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:18:14.832921+08:00, lastProcessTime=2023-05-15T14:18:04.829668+08:00 2023-05-15 14:18:14.874 INFO 11 --- [alarm-detect-7,,aebd5d05cc96] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131474 ~ 1684131484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131480=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:14.903 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:18:14.832921+08:00 2023-05-15 14:18:15.145 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:15.146 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:16.062 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,29850a8585f54cde,c5a1910e1738] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.11, traceId=29850a8585f54cde, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:18:16.069 INFO 11 --- [http-nio-0.0.0.0-8080-exec-62,29850a8585f54cde,c5a1910e1738] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.11, traceId=29850a8585f54cde, duration=7 ms] 2023-05-15 14:18:16.281 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:16.284 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:16.286 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:16.313 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:16.456 INFO 11 --- [pool-ocp-schedules-20,,a06a89e4eff7] c.o.o.m.service.SeriesIdCleanService : HostIpSize=5, clusterSize=2, obproxyClusterSize=1 2023-05-15 14:18:17.271 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:18.146 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:18.147 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:18.279 INFO 11 --- [pool-overview-container-4,,a81182718fef] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:18.538 INFO 11 --- [pool-scanner--15980200-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:18:19.844 INFO 11 --- [pool-ocp-schedules-17,,5f5b72f0222b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:18:20.186 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:20.0, nextRun=2023-05-15 14:18:20.0, currentDate=Mon May 15 14:18:20 CST 2023 2023-05-15 14:18:20.187 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=6, name=Collect all cluster compaction info, templateName=CollectClusterCompactionSchedule, scheduleRule=20 * * * * *, lastRunTime=2023-05-15 14:17:20.0, nextRunTime=2023-05-15 14:18:20.0} 2023-05-15 14:18:20.190 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all cluster compaction info, nodeCount=1, taskCount=1 2023-05-15 14:18:20.190 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:20.190 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all cluster compaction info, taskDefinitionId=6, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:20.614 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132068, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:20.616 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132068, name=Collect compaction for all clusters, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.CollectAllClustersCompactionTask, seriesId=0, startTime=2023-05-15T14:18:20.612+08:00, endTime=null} 2023-05-15 14:18:20.636 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d47675ab31de] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132068, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116445, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:20.612+08:00, sub_task_instance_id=132068}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:20.638 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d47675ab31de] c.o.o.s.t.b.c.AbstractAllClusterTask : Collect compaction for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:18:20.639 INFO 11 --- [ForkJoinPool-66116-worker-1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:18:20.639 INFO 11 --- [ForkJoinPool-66116-worker-2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2023-05-15 14:18:20.644 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,d47675ab31de] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132068, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:21.148 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:21.149 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:21.337 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116445, status:SUCCESS 2023-05-15 14:18:21.339 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116445, status:SUCCESS 2023-05-15 14:18:21.924 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_INDEX_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:21.935 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PLAN_CHANGE_DETECTOR updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:21.937 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService SQL_PERF_DIAGNOSER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:23.808 INFO 11 --- [pool-scanner--15980201-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:18:23.829 INFO 11 --- [pool-scanner--15980202-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:18:23.854 INFO 11 --- [pool-scanner--15980203-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:18:23.875 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:23.882 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:23.920 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:23.923 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:23.935 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:23.938 INFO 11 --- [pool-ocp-schedules-6,,0de0c0eb078f] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:24.149 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:24.150 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:24.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:18:24.833 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:18:24.833102+08:00, lastProcessTime=2023-05-15T14:18:14.832921+08:00 2023-05-15 14:18:24.929 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:18:24.833102+08:00 2023-05-15 14:18:24.961 INFO 11 --- [alarm-detect-4,,af15708efbec] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131434 ~ 1684131494 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=389877.988, values={1684131480=389877.988}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:18:24.963 INFO 11 --- [alarm-detect-4,,af15708efbec] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1684131434 ~ 1684131494 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=994675.358, values={1684131480=994675.358}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:24.966 INFO 11 --- [alarm-detect-12,,1f87998ee227] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131484 ~ 1684131494 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131490=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:24.966 INFO 11 --- [alarm-detect-4,,af15708efbec] c.o.o.s.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=sys-2, alarm_template_id=0:ob_cluster=obcluster-1:tenant_name=test_tenant-2] 2023-05-15 14:18:25.346 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:25.0, nextRun=2023-05-15 14:18:25.0, currentDate=Mon May 15 14:18:25 CST 2023 2023-05-15 14:18:25.347 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=9, name=Check host status, templateName=HostCheck, scheduleRule=25 * * * * *, lastRunTime=2023-05-15 14:17:25.0, nextRunTime=2023-05-15 14:18:25.0} 2023-05-15 14:18:25.358 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Check host status, nodeCount=1, taskCount=1 2023-05-15 14:18:25.358 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:25.359 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Check host status, taskDefinitionId=9, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:25.684 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132069, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:25.685 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132069, name=Check host exporter status, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CheckExporterTask, seriesId=0, startTime=2023-05-15T14:18:25.681+08:00, endTime=null} 2023-05-15 14:18:25.708 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132069, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116446, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:25.681+08:00, sub_task_instance_id=132069}, listMap={host_ids=[1, 2, 3, 4, 5]}}, executor=10.xxx.xx.10 2023-05-15 14:18:25.748 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=1, ip=10.xxx.xx.10] desired exporter count: 6, actual exporter count: 6 2023-05-15 14:18:25.749 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=2, ip=10.xxx.xx.11] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:18:25.750 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=3, ip=10.xxx.xx.12] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:18:25.751 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=4, ip=10.xxx.xx.13] desired exporter count: 8, actual exporter count: 8 2023-05-15 14:18:25.752 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.s.t.b.host.CheckExporterTask : Check exporter status: host [id=5, ip=10.xxx.xx.14] desired exporter count: 3, actual exporter count: 3 2023-05-15 14:18:25.758 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,e68ff176f680] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132069, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:26.284 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService MC_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:26.287 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService CMDB_SYNCER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:26.290 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService TASK_MANAGER updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:26.315 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService RATE_LIMIT updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:26.439 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116446, status:SUCCESS 2023-05-15 14:18:26.440 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116446, status:SUCCESS 2023-05-15 14:18:27.076 INFO 11 --- [pool-ocp-schedules-14,,95eb46d5e875] c.o.o.s.task.scheduled.ClusterSchedules : Check host status schedule. 2023-05-15 14:18:27.077 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:18:27.077 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:18:27.077 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:18:27.077 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:18:27.077 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:18:27.084 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:18:27.084 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:18:27.084 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:18:27.084 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:18:27.085 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:18:27.111 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/time, params:null 2023-05-15 14:18:27.111 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/time, params:null 2023-05-15 14:18:27.112 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/time, params:null 2023-05-15 14:18:27.112 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/time, params:null 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=1, ip=10.xxx.xx.10, timestamp=2023-05-15T14:18:27.113455461+08:00 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=3, ip=10.xxx.xx.12, timestamp=2023-05-15T14:18:27.113647563+08:00 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:18:27.114+08:00 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:18:27.114+08:00 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=5, ip=10.xxx.xx.14, timestamp=2023-05-15T14:18:27.114300679+08:00 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/time, params:null 2023-05-15 14:18:27.114 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:18:27.114+08:00 2023-05-15 14:18:27.115 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=4, ip=10.xxx.xx.13, timestamp=2023-05-15T14:18:27.114542139+08:00 2023-05-15 14:18:27.115 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:18:27.115+08:00 2023-05-15 14:18:27.117 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent check result. id=2, ip=10.xxx.xx.11, timestamp=2023-05-15T14:18:27.116535643+08:00 2023-05-15 14:18:27.117 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:18:27.117+08:00 2023-05-15 14:18:27.126 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:18:27.114+08:00 and status AVAILABLE 2023-05-15 14:18:27.129 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:18:27.115+08:00 and status AVAILABLE 2023-05-15 14:18:27.131 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:18:27.114+08:00 and status AVAILABLE 2023-05-15 14:18:27.133 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:18:27.114+08:00 and status AVAILABLE 2023-05-15 14:18:27.134 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:18:27.117+08:00 and status AVAILABLE 2023-05-15 14:18:27.151 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:27.152 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:27.175 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.10], timeoutMillis=20000 2023-05-15 14:18:27.175 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.13], timeoutMillis=20000 2023-05-15 14:18:27.175 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.11], timeoutMillis=20000 2023-05-15 14:18:27.176 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.14], timeoutMillis=20000 2023-05-15 14:18:27.176 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.xxx.xx.12], timeoutMillis=20000 2023-05-15 14:18:27.219 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131507 0 0 , error= 2023-05-15 14:18:27.219 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131507 0 0 , error= 2023-05-15 14:18:27.220 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2023-05-15T14:18:27.220+08:00 2023-05-15 14:18:27.220 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2023-05-15T14:18:27.220+08:00 2023-05-15 14:18:27.219 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131507 0 0 , error= 2023-05-15 14:18:27.222 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131507 0 0 , error= 2023-05-15 14:18:27.232 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1684131507 0 0 , error= 2023-05-15 14:18:27.232 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2023-05-15T14:18:27.221+08:00 2023-05-15 14:18:27.238 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2023-05-15T14:18:27.237+08:00 2023-05-15 14:18:27.238 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2023-05-15T14:18:27.237+08:00 2023-05-15 14:18:27.239 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 1 last available time 2023-05-15T14:18:27.220+08:00 and status AVAILABLE 2023-05-15 14:18:27.239 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 4 last available time 2023-05-15T14:18:27.220+08:00 and status AVAILABLE 2023-05-15 14:18:27.242 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 5 last available time 2023-05-15T14:18:27.237+08:00 and status AVAILABLE 2023-05-15 14:18:27.242 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2 last available time 2023-05-15T14:18:27.237+08:00 and status AVAILABLE 2023-05-15 14:18:27.244 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=1 2023-05-15 14:18:27.247 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=2 2023-05-15 14:18:27.247 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=4 2023-05-15 14:18:27.247 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=5 2023-05-15 14:18:27.253 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 3 last available time 2023-05-15T14:18:27.221+08:00 and status AVAILABLE 2023-05-15 14:18:27.255 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:18:27.256 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:18:27.256 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent process running info: hostId=3 2023-05-15 14:18:27.257 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:18:27.258 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:18:27.267 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:18:27.275 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:27.290 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:18:27.291 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:18:27.292 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:18:27.292 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:18:27.294 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent), params:null 2023-05-15 14:18:27.338 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2023-05-15 14:18:27.347 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2023-05-15 14:18:27.349 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2023-05-15 14:18:27.355 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2023-05-15 14:18:27.361 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2023-05-15 14:18:27.383 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2023-05-15 14:18:27.389 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2023-05-15 14:18:27.391 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2023-05-15 14:18:27.393 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:18:27.393 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2023-05-15 14:18:27.401 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2023-05-15 14:18:27.413 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:18:27.422 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:18:27.426 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:18:27.429 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent), params:null 2023-05-15 14:18:27.435 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2023-05-15 14:18:27.444 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2023-05-15 14:18:27.459 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.14:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:18:27.475 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2023-05-15 14:18:27.479 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2023-05-15 14:18:27.486 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2023-05-15 14:18:27.486 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2023-05-15 14:18:27.488 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2023-05-15 14:18:27.495 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2023-05-15 14:18:27.497 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.13:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:18:27.501 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.11:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:18:27.505 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2023-05-15 14:18:27.506 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.12:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:18:27.509 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2023-05-15 14:18:27.515 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2023-05-15 14:18:27.518 INFO 11 --- [pool-Check host status29,,d7d532625c73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2023-05-15 14:18:27.531 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.xxx.xx.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd), params:null 2023-05-15 14:18:27.550 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2023-05-15 14:18:27.562 INFO 11 --- [pool-Check host status7,,3fdc87203324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2023-05-15 14:18:27.564 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2023-05-15 14:18:27.564 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2023-05-15 14:18:27.571 INFO 11 --- [ocp-updater-0,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService PERF_SQL_ROLLUP updating status on 10.xxx.xx.10:8080 2023-05-15 14:18:27.576 INFO 11 --- [pool-Check host status4,,1732a826fd8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2023-05-15 14:18:27.578 INFO 11 --- [pool-Check host status21,,058711a4a930] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2023-05-15 14:18:27.614 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2023-05-15 14:18:27.626 INFO 11 --- [pool-Check host status6,,6522ae3da463] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2023-05-15 14:18:27.642 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.xxx.xx.11, originalStatus=ONLINE 2023-05-15 14:18:27.642 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.xxx.xx.10, originalStatus=ONLINE 2023-05-15 14:18:27.642 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.xxx.xx.13, originalStatus=ONLINE 2023-05-15 14:18:27.642 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.xxx.xx.12, originalStatus=ONLINE 2023-05-15 14:18:27.642 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.xxx.xx.14, originalStatus=AVAILABLE 2023-05-15 14:18:27.647 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2023-05-15 14:18:27.648 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2023-05-15 14:18:27.649 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2023-05-15 14:18:27.649 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2023-05-15 14:18:27.649 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2023-05-15 14:18:27.671 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.13:62888/api/v1/agent/status, params:null 2023-05-15 14:18:27.674 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.14:62888/api/v1/agent/status, params:null 2023-05-15 14:18:27.677 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.11:62888/api/v1/agent/status, params:null 2023-05-15 14:18:27.679 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.10:62888/api/v1/agent/status, params:null 2023-05-15 14:18:27.679 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.xxx.xx.12:62888/api/v1/agent/status, params:null 2023-05-15 14:18:27.692 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.13 2023-05-15 14:18:27.692 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:18:27.692 INFO 11 --- [pool-Check host status27,,41866df3da3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.13 2023-05-15 14:18:27.695 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.14 2023-05-15 14:18:27.695 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:18:27.695 INFO 11 --- [pool-Check host status5,,9ee99d125f24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.14 2023-05-15 14:18:27.698 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.11 2023-05-15 14:18:27.698 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:18:27.698 INFO 11 --- [pool-Check host status11,,d78926e82352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.11 2023-05-15 14:18:27.700 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.12 2023-05-15 14:18:27.700 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:18:27.700 INFO 11 --- [pool-Check host status23,,92462f7f4668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.12 2023-05-15 14:18:27.705 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.xxx.xx.10 2023-05-15 14:18:27.705 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:18:27.705 INFO 11 --- [pool-Check host status20,,7759a13f3154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.xxx.xx.10 2023-05-15 14:18:28.280 INFO 11 --- [pool-overview-container-8,,ad7986609153] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:18:28.281 INFO 11 --- [pool-overview-container-8,,ad7986609153] c.o.o.s.a.service.OcpAlarmSearchService : sortedAlarm size=40, alarmHistory size=496 2023-05-15 14:18:28.524 INFO 11 --- [pool-scanner--15980301-1,,] c.o.o.m.store.RollupMetricDataStore : fetchFromDb, nodesCountFromDb=14, nodesFilledCount=0 2023-05-15 14:18:29.866 INFO 11 --- [pool-ocp-schedules-19,,d52870d25aee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2023-05-15 14:18:30.152 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:30.153 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:30.334 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,499fa256fba549b8,a59a6e0f30b4] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba&VersionOnly=true, client=10.xxx.xx.10, traceId=499fa256fba549b8, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=admin,alibaba,,,true,] 2023-05-15 14:18:30.336 INFO 11 --- [http-nio-0.0.0.0-8080-exec-63,499fa256fba549b8,a59a6e0f30b4] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.10, traceId=499fa256fba549b8, duration=2 ms] 2023-05-15 14:18:30.525 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:17:31.0, nextRun=2023-05-15 14:18:30.0, currentDate=Mon May 15 14:18:30 CST 2023 2023-05-15 14:18:30.525 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=16, name=Sync tenant information, templateName=TenantSyncSchedule, scheduleRule=30 0/1 * * * ?, lastRunTime=2023-05-15 14:17:31.0, nextRunTime=2023-05-15 14:18:30.0} 2023-05-15 14:18:30.532 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Sync tenant information, nodeCount=1, taskCount=2 2023-05-15 14:18:30.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:30.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:30.533 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Sync tenant information, taskDefinitionId=16, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:30.545 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2023-05-15 14:18:01.0, nextRun=2023-05-15 14:18:30.0, currentDate=Mon May 15 14:18:30 CST 2023 2023-05-15 14:18:30.545 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=34, name=Collect all tenants major compactions, templateName=CollectTenantCompactionSchedule, scheduleRule=*/30 * * * * ?, lastRunTime=2023-05-15 14:18:01.0, nextRunTime=2023-05-15 14:18:30.0} 2023-05-15 14:18:30.550 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.model.TaskInstanceEntity : generate tasks, templateName=Collect all tenants major compactions, nodeCount=1, taskCount=1 2023-05-15 14:18:30.550 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2023-05-15 14:18:30.550 INFO 11 --- [pool-task-coordinator1,fb27d19334114ed7,f8c159a0a6ca] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name=Collect all tenants major compactions, taskDefinitionId=34, state=RUNNING, operation=EXECUTE} 2023-05-15 14:18:30.716 INFO 11 --- [pool-ocp-schedules-20,,200c92d47ced] c.o.o.s.a.service.OcpAlarmSearchService : alarms load from database, alarm count=2 2023-05-15 14:18:30.769 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132070, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:30.771 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132070, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=0, startTime=2023-05-15T14:18:30.767+08:00, endTime=null} 2023-05-15 14:18:30.781 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132071, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:30.783 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132071, name=Sync all tenant information, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=1, startTime=2023-05-15T14:18:30.779+08:00, endTime=null} 2023-05-15 14:18:30.788 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132070, context=Context{parallelIdx=0, stringMap={prohibit_rollback=false, task_instance_id=116447, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:30.767+08:00, sub_task_instance_id=132070}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:18:30.791 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132072, operation:EXECUTE, state: RUNNING 2023-05-15 14:18:30.793 INFO 11 --- [pool-subtask-coordinator2,,] c.o.o.c.t.e.c.w.s.ReadySubtaskWorker : Submit ready task, entity=SubtaskInstanceOverview{id=132072, name=Collect all tenants mini compactions, state=RUNNING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantCompactionTask, seriesId=0, startTime=2023-05-15T14:18:30.789+08:00, endTime=null} 2023-05-15 14:18:30.802 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132071, context=Context{parallelIdx=1, stringMap={prohibit_rollback=false, task_instance_id=116447, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:30.779+08:00, sub_task_instance_id=132071}, listMap={cluster_ids=[1, 2]}}, executor=10.xxx.xx.10 2023-05-15 14:18:30.812 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a113e9087ca4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=132072, context=Context{parallelIdx=-1, stringMap={prohibit_rollback=false, task_instance_id=116448, task_operation=execute, latest_execution_start_time=2023-05-15T14:18:30.789+08:00, sub_task_instance_id=132072}, listMap={}}, executor=10.xxx.xx.10 2023-05-15 14:18:30.813 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a113e9087ca4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2023-05-15 14:18:30.819 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.820 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.820 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.820 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.827 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.827 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.828 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:18:30.831 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:18:30.843 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.843 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.843 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.843 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.844 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.844 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.844 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:30.844 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.845 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.853 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.853 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.853 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.853 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.854 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:18:30.854 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.854 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.854 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.854 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.855 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.855 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.855 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.855 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.855 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2023-05-15 14:18:30.856 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.856 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.856 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:30.856 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201900144091, broadcast scn:1684087201900144091, saved frozen scn:1684087201900144091 2023-05-15 14:18:30.858 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.859 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.859 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.859 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.859 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.859 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:18:30.859 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087203911369536, broadcast scn:1684087203911369536, saved frozen scn:1684087203911369536 2023-05-15 14:18:30.860 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.861 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.861 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.861 INFO 11 --- [pool-3249019-thread-1,fb27d19334114ed7,c2dac676e4db] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:meta_tenant 2023-05-15 14:18:30.861 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2023-05-15 14:18:30.861 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087201760149760, broadcast scn:1684087201760149760, saved frozen scn:1684087201760149760 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-4,fb27d19334114ed7,e8691dcf8a76] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:sys 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.862 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2023-05-15 14:18:30.863 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2023-05-15 14:18:30.864 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1684087200398187132, broadcast scn:1684087200398187132, saved frozen scn:1684087200398187132 2023-05-15 14:18:30.864 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.864 INFO 11 --- [pool-3249019-thread-3,fb27d19334114ed7,f7bf35d97ab2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:oms_tenant 2023-05-15 14:18:30.864 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.865 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.865 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.865 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2023-05-15 14:18:30.865 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:30.866 INFO 11 --- [pool-3249019-thread-2,fb27d19334114ed7,e363ae034289] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:ocpcluster, tenant:monitor_tenant 2023-05-15 14:18:30.867 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683136804641344843, broadcast scn:1683136804641344843, saved frozen scn:1683136804641344843 2023-05-15 14:18:30.867 INFO 11 --- [pool-3249019-thread-6,fb27d19334114ed7,a600107cb22f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:test_tenant 2023-05-15 14:18:30.868 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1683741602011070816, broadcast scn:1683741602011070816, saved frozen scn:1683741602011070816 2023-05-15 14:18:30.869 INFO 11 --- [pool-3249019-thread-5,fb27d19334114ed7,602277443e84] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obcluster, tenant:sys 2023-05-15 14:18:30.869 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.869 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.869 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a113e9087ca4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2023-05-15 14:18:30.870 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:30.873 INFO 11 --- [pool-subtask-executor26,fb27d19334114ed7,a113e9087ca4] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132072, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:30.875 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.875 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.875 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:30.882 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.882 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.882 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:30.884 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.884 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.884 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:18:30.898 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.898 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.898 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:18:30.899 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.899 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.900 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:30.907 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.907 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.908 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:18:30.915 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.916 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.916 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:30.918 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.918 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.919 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:18:30.927 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:18:30.929 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.tenant.TenantSyncServiceImpl : [OCP606201] detect obsolete tenants, delete in ocp: [] 2023-05-15 14:18:30.947 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:30.947 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.948 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.948 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.948 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.954 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.954 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.955 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:30.956 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:30.957 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.958 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.958 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.958 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.963 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.963 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.964 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:30.965 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:30.965 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.965 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.965 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.965 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.967 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.967 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.967 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:30.981 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:30.982 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.983 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.983 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.984 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.985 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:30.985 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.986 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.986 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.986 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:30.987 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:30.987 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.988 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:30.988 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:30.988 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:30.989 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:31.000 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.001 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.003 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.003 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.003 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:31.023 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.024 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.024 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.024 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.025 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.025 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.026 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.027 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.027 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:31.029 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.029 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.030 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:31.040 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.040 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.041 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.041 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.041 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.042 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.043 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:31.049 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.050 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.051 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.051 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.051 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.056 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.056 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.058 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:31.059 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2023-05-15 14:18:31.067 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.067 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.068 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.070 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.070 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.070 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2023-05-15 14:18:31.071 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:18:31.084 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.084 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.084 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.085 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.085 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.086 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.086 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.086 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2023-05-15 14:18:31.087 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2023-05-15 14:18:31.090 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.xxx.xx.10 2023-05-15 14:18:31.103 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.104 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.105 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.106 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.106 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:31.108 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.108 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.108 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:31.111 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.111 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.112 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:31.124 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.126 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2023-05-15 14:18:31.126 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.126 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.127 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2023-05-15 14:18:31.127 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.129 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2023-05-15 14:18:31.130 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:31.130 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.131 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2023-05-15 14:18:31.135 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.138 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2023-05-15 14:18:31.138 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.138 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.138 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2023-05-15 14:18:31.143 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2023-05-15 14:18:31.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2023-05-15 14:18:31.145 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:18:31.147 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.149 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2023-05-15 14:18:31.149 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:31.150 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:31.150 INFO 11 --- [pool-subtask-executor25,fb27d19334114ed7,bd94f5336a7a] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132071, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:31.150 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2023-05-15 14:18:31.163 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.xxx.xx.10, userWhitelist=% 2023-05-15 14:18:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2023-05-15 14:18:31.165 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2023-05-15 14:18:31.166 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.s.t.b.t.SyncAllTenantInfoTask : [OCP606] sync tenants success, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:18:31.170 INFO 11 --- [pool-subtask-executor27,fb27d19334114ed7,5a2d8689ca27] c.o.o.c.m.repo.task.SubtaskAccessor : Set state for subtask: 132070, operation:EXECUTE, state: SUCCESSFUL 2023-05-15 14:18:31.172 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,19c1202609ac4007,56fff8bc70a6] c.o.o.server.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxy&VersionOnly=true, client=10.xxx.xx.13, traceId=19c1202609ac4007, method=SuccessResult com.oceanbase.ocp.server.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxy,true,] 2023-05-15 14:18:31.176 INFO 11 --- [http-nio-0.0.0.0-8080-exec-4,19c1202609ac4007,56fff8bc70a6] c.o.o.server.trace.RequestTracingAspect : API OK: [GET /services client=10.xxx.xx.13, traceId=19c1202609ac4007, duration=4 ms] 2023-05-15 14:18:31.539 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116447, status:SUCCESS 2023-05-15 14:18:31.540 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116447, status:SUCCESS 2023-05-15 14:18:31.550 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:116448, status:SUCCESS 2023-05-15 14:18:31.551 INFO 11 --- [pool-task-coordinator2,adea1912c501423b,914674b9c11f] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:116448, status:SUCCESS 2023-05-15 14:18:32.068 INFO 11 --- [pool-ocp-schedules-3,,389faa6e0871] c.o.o.s.a.service.TaskRepositoryService : heartbeat done, affectRow=2, taskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:18:32.068 INFO 11 --- [pool-ocp-schedules-3,,389faa6e0871] c.o.o.s.a.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=2, heartbeatCount=2, currentTaskUuidList=[b46f8c87-3863-4ebd-bcea-991422baa7ff, a803546b-430b-4932-a7ca-e822f2a2c472] 2023-05-15 14:18:32.114 INFO 11 --- [pool-ocp-schedules-10,,27e094a4884b] c.o.o.s.task.scheduled.ClusterSchedules : Check cluster status schedule. 2023-05-15 14:18:32.116 INFO 11 --- [pool-ocp-schedules-10,,27e094a4884b] c.o.o.s.t.b.c.AbstractAllClusterTask : Check status for all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:18:32.122 INFO 11 --- [pool-ocp-schedules-5,,73c12d6aeb2c] c.o.o.s.task.scheduled.ClusterSchedules : Check standby cluster info schedule. 2023-05-15 14:18:32.124 INFO 11 --- [pool-ocp-schedules-5,,73c12d6aeb2c] c.o.o.s.t.b.c.AbstractAllClusterTask : Check all standby clusters sync delay time executed completed, tryCount=2, successCount=0 2023-05-15 14:18:32.135 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.136 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.136 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.136 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.137 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.137 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.137 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.138 INFO 11 --- [ForkJoinPool-66117-worker-2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.1.0.0 2023-05-15 14:18:32.140 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.140 INFO 11 --- [pool-ocp-schedules-17,,b0596972ba85] c.o.o.s.task.scheduled.ClusterSchedules : Sync cluster info schedule. 2023-05-15 14:18:32.141 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.141 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.142 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.144 INFO 11 --- [pool-ocp-schedules-17,,b0596972ba85] c.o.o.s.t.b.c.AbstractAllClusterTask : Sync all clusters executed completed, tryCount=2, successCount=0 2023-05-15 14:18:32.146 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.146 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.146 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.150 INFO 11 --- [ForkJoinPool-66117-worker-1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.1.0.0 2023-05-15 14:18:32.167 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.167 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.167 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.167 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.173 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.176 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.176 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.176 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.176 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.177 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.177 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:18:32.181 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.181 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.182 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:18:32.225 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:32.225 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.226 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.226 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.226 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.227 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.227 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.227 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:18:32.228 WARN 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2023-05-15 14:18:32.235 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.235 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.235 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:18:32.236 INFO 11 --- [ForkJoinPool-66119-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.10:2982, status=ACTIVE, startServiceTime=1683530134960216, active=true 2023-05-15 14:18:32.236 INFO 11 --- [ForkJoinPool-66119-worker-2,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.10:2982, stopTime=0, stopped=false 2023-05-15 14:18:32.236 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:18:32.250 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.250 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T15:15:35+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:18:32.256 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:32.257 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.257 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.257 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.258 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.263 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.263 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.263 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2023-05-15 14:18:32.267 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.267 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.268 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2023-05-15 14:18:32.268 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.xxx.xx.10, port=2982, sqlPort=2981, version=4.1.0.0_100000172023031416-6d4641c7bcd0462f0bf3faed011803c087ea1705(Mar 14 2023 16:53:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-08T07:15:34.960216Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.270 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.11:2982, status=ACTIVE, startServiceTime=1681989951900812, active=true 2023-05-15 14:18:32.270 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.11:2982, stopTime=0, stopped=false 2023-05-15 14:18:32.270 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.12:2982, status=ACTIVE, startServiceTime=1683708604272665, active=true 2023-05-15 14:18:32.270 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.12:2982, stopTime=0, stopped=false 2023-05-15 14:18:32.271 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer active, observer=10.xxx.xx.13:2982, status=ACTIVE, startServiceTime=1681989952296456, active=true 2023-05-15 14:18:32.271 INFO 11 --- [ForkJoinPool-66119-worker-1,,] com.oceanbase.ocp.obsdk.util.ObSdkUtils : check observer stopped, observer=10.xxx.xx.13:2982, stopTime=0, stopped=false 2023-05-15 14:18:32.271 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:18:32.281 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=ocpcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:32.282 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.282 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.282 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.282 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.284 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:10.xxx.xx.10, sql port:2981 2023-05-15 14:18:32.284 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.284 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.289 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.289 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:18:32.294 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=ocpcluster, obClusterId=1 2023-05-15 14:18:32.294 INFO 11 --- [ForkJoinPool-66119-worker-2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=ocpcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"ocpcluster","ObClusterId":1,"ObRegion":"ocpcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.10:2982","role":"LEADER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131336320000}, createTime=2023-05-08T15:36:09+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=4) 2023-05-15 14:18:32.302 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.xxx.xx.13, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:52.296456Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.302 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:18:32.315 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone2, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.315 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T16:50:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:18:32.328 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=7, ip=10.xxx.xx.12, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2023-05-10T08:50:04.272665Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.328 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2023-05-15 14:18:32.342 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone1, idcId=2, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.342 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T19:25:52+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z} 2023-05-15 14:18:32.358 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.xxx.xx.11, port=2982, sqlPort=2981, version=4.1.0.0_100000202023040520-0765e69043c31bf86e83b5d618db0530cf31b707(Apr 5 2023 20:26:14), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2023-04-20T11:25:51.900812Z, stopTime=1970-01-01T00:00Z}, oldStatus=RUNNING, newStatus=RUNNING 2023-05-15 14:18:32.377 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, userId=200, clusterName=obcluster, tenantName=sys, dbUser=ocp_monitor 2023-05-15 14:18:32.378 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.379 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.379 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.379 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.384 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:127.0.0.1, sql port:2888 2023-05-15 14:18:32.384 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2023-05-15 14:18:32.385 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2023-05-15 14:18:32.405 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=obcluster, obClusterId=1 2023-05-15 14:18:32.405 INFO 11 --- [ForkJoinPool-66119-worker-1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=obcluster, obClusterId=1, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, syncStatus=VALID, rootserverJson={"ObCluster":"obcluster","ObClusterId":1,"ObRegion":"obcluster","ObRegionId":1,"ReadonlyRsList":[],"RsList":[{"address":"10.xxx.xx.11:2982","role":"LEADER","sql_port":2981},{"address":"10.xxx.xx.13:2982","role":"FOLLOWER","sql_port":2981},{"address":"10.xxx.xx.12:2982","role":"FOLLOWER","sql_port":2981}],"Type":"PRIMARY","timestamp":1684131346904000}, createTime=2023-05-09T14:00:35+08:00, creator=admin, obVersion=4.1.0.0, tenantCount=2) 2023-05-15 14:18:33.153 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.xxx.xx.10:8080 refreshes policies... 2023-05-15 14:18:33.154 INFO 11 --- [pool-9-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2023-05-15 14:18:34.785 INFO 11 --- [alarm-task-1,,38cd5ad3a0df] c.o.o.s.alarm.core.task.AlarmDetectTask : alarm detect start 2023-05-15 14:18:34.831 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process alarm start, currentTimestamp=2023-05-15T14:18:34.830865+08:00, lastProcessTime=2023-05-15T14:18:24.833102+08:00 2023-05-15 14:18:34.856 INFO 11 --- [alarm-detect-13,,e1eea080d8cb] c.o.o.s.a.core.detect.MetricDetector : alarmType tenant_active_memstore_percent_over_threshold detect time range 1684131494 ~ 1684131504 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={active_memstore_percent > 110=[ExpressionEvaluateResult(matchValue=110.0, metric=active_memstore_percent, match=true, value=246.401, values={1684131500=246.401}, valueCount=1, labels={app=OB, host=10.xxx.xx.13, host_name=obdb3, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, obzone=zone3, svr_ip=10.xxx.xx.13, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:34.864 INFO 11 --- [alarm-detect-3,,9c00d45aebc7] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131444 ~ 1684131504 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=389877.938, values={1684131480=389877.938}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1, tenant_name=sys})]}, matched=true, message=null, level=2) 2023-05-15 14:18:34.865 INFO 11 --- [alarm-detect-3,,9c00d45aebc7] c.o.o.s.a.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1684131444 ~ 1684131504 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=994675.107, values={1684131480=994675.107}, valueCount=1, labels={app=OB, ob_cluster_id=1, ob_cluster_name=obcluster, ob_tenant_id=1002, tenant_name=test_tenant})]}, matched=true, message=null, level=2) 2023-05-15 14:18:34.893 INFO 11 --- [alarm-task-2,,b707bf68cf49] c.o.o.s.a.core.process.AlarmProcessor : process done, currentTimestamp=2023-05-15T14:18:34.830865+08:00